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ABSTRACT 

This, the second General Information Yearbook, is 
designed to assist the reader in understanding Assessment findings. 
Report 03/04-6IY relates especially to the learning areas assessed 
during the third and fourth years of the project: social studies, 
music, mathematics, and science, it presents an overview of National 
Assessment, describing its purpose, history, and methodology, and 
then goes on to devote a chapter to each of the major steps in the 
intricate research/dissemination process. The chapters are organized 
to tell how the assessment is carried out and subsequently how its 
data are reported and used. National Assessment's (NAEP) principal 
objective is to assess the changes over time in educational 
achievement of four age groups in the U.S. In addition to the 
national results for each age group, results aro reported within each 
age level for certain major subpopulations, such as geographic region 
and sex. To do so, NAEP had developed two baseline measures: (1) an 
estimate of the percentage of persons in each of the populations who 
gave a certain response to a specific exercise, and (2) an estimate 
of the difference between the performance of a subpopulation and the 
performance of the total age population. Measures of change will 
reflect comparisons of these statistics over time. (Author/JH) 
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PREFACE 



This is the second General Information Year- 
book published by the National Assessment 
of Educational Progress (NAEP). It is design- 
ed to assist the reader in understanding 
Assessment findings. Report 03/04.GIY re- 
lates especially to the learning areas assessed 
during the third and fourth years of the 
project: social studies, music, mathematics 
and science. It presents an overview of 
National Assessment, describing its purpose, 
history and methodology, and then goes on to 
devote a chapter to each of the mjgor steps in 
the intricate research/dissemination process. 

The chapters of the General Information 
Yearbook are organized to tell how the 
assessment is carried out and subsequently 
how its data are reported and used. The 
appendixes provide a more detailed descrip- 
tion of procedures, groups, reporting cate- 
gories and persons involved. For more tech- 
nical descriptions of National Assessment 
procedures, see our mo;->ographs entitled The 
National Assessment Approach to Exercise 
Development (1970) and The National Assess- 
ment Approach to Sampling (1974). 



Repotting and Numbering System 

For the first assessment year, 1969—70, re- 
sults were reported primarily in statistical 
volumes numbered chronologically. National 
results for science, for example, are found in 
Report 1, citizenship in Report 2 and writing 
in Report 3. Results are reported by groups in 
subsequent reports. 

Reading and literature, the learning areas 
assessed in Year 02, 1970—71, are reported 



by themes or topics. At that time, a new 
numbering system was instituted which indi- 
cates the assessment year, the learning area 
and the type of data that will be found inside. 
The new report number consists of three 
parts. The first part is a two-digit number 
referring to the assessment year— for example, 
02 for the second assessment year. The 
second part is an initial letter or letters for the 
learning areas as follows: A for art, COD for 
career and occupational development, C for 
citizenship, L for literature, MA for mathe- 
matics, MU for music, R for reading, S for 
science, SS for social studies and W for 
writing. 

The third part is a two-digit number indi- 
cating the content of the volume as follows: 

00 = a summary or overview of a 
learning area 

01—19 * results for a theme or topic 
within a learning area 

20 « released exercise or statis- 
tical report 

21—29 = research or special reports 
as applicable 

30 « a digest or condensation of 
findings 

For example, the Reading Summary is Report 

02- R-OO, the Social Studies Overview is 

03- SS-OO, the statistical report containing the 
exercises with results for mathematics will be 
numbered 04-MA-20. 
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How to Secure Repot ts 

The front inside cover of this report lists all of 
the reports issued by National Assessment to 
date. Although limited copies are kept at the 
Denver office for consultants, committee 
members and media representatives, NAEP 



rr^ports should be ordered directly from the 
Superintendent of Documents, U.S. Govern- 
ment Printing Office, Washington, D.C. 
20402. When ordering, check must be en- 
closed as well as full title, e.g.. National 
Assessment of Educational Progress, Report 
No. 3, Writing - National Results. 
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CHAPTER 1 
OVERVIEW OF NATIONAL ASSESSMENT 



History and Purpose 

When the United States Office of Education 
was founded in 1867, one charge set before 
its commissioner was to determine the na- 
tion's progress in education. That century-old 
charge is only now being answered by the 
National Assessment of Educational Progress 
(NAKP), a project of the Education Commis- 
sion of the States. 

By the early 1960s, the average annual ex- 
penditure of public funds for the formal 
education of young Americans was $30 bil- 
lion," Yet criticism of the educational system 
abounded. Defenders of the educational 
establishment found it increasingly difficult 
to provide evidence that the schools were 
satisfactorily meeting the educational needs 
of a modem, technological society. The only 
readily available measures of educational qual- 
ity resulting from this public investment of 
funds were input measures such as teacher- 
student ratios and per-pupil expenditures. The 
tenuous assumption was made that the qual- 
ity of educational outcomei^what students 
do or do not know and can or cannot do— was 
directly related to the quality of the inputs to 
the educational system. There has been no 
conclusive empirical evidence to support this 
assumption. The typical standardized achieve- 
ment tests administered by schools or states 
provided scores whereby one student could be 
compared with other students. Such informa- 
tion was useful in categorizing students; 
however, it provided little information about 
what students were or were not actually 



Based on United States Census Bureau data. Actual 
amounts expended were: 1960. $24.7 billion; 1962, 
$29.4 billion; 1964. $35.9 billion. 



learning. There had been no direct assessment 
of educational outcomes. 

This insufficiency of information became the 
concern of Dr. Francis Keppel, United States 
commissioner of education (1962-65). He 
initiated a series of conferences to explore 
ways to provide the necessary information. In 
1964, as a result of these conferences, John 
W. Gardner, president of the Carnegie Corpo- 
ration, asked a distinguished group of educa- 
tors and other concerned persons to form the 
Exploratory Committee on Assessing the 
Progress of Education (ECAPE). This com- 
mittee, chaired by Dr. Ralph W. Tyler who 
had been involved since the earliest confer- 
ences, was to examine the possibility of 
conducting an assessment of educational 
attainments on a national basis. 

After much study, ECAPE decided that it was 
feasible to inaugurate an assessment project to 
fill the information gap regarding the quality 
of educational outcomes by a periodic assess- 
ment of the knowledges, understandings, 
skills and attitudes in 10 learning areas^ at 
four age levels (9, 13, 17 and young adult- 
ages 26-35). This project-named the 
Committee on Assessing the Progress of Edu- 
cation (CAPE)-began its charge under the 
auspices of the Carnegie Corporation by 
assessing the learning areas of science, citizen- 
ship and writing in the spring of 1969. Later 
that same year, the project came under the 
auspices of the Education Commission of the 
States. Funding and monitoring were trans- 
ferred to the United States Office of Educa- 



Art, career and occupational development, citi- 
zenship, literature, mathematics, music, reading, sci- 
ence, social studies and writing. 
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tion» and the project was renamed the 
National Assessment of Educational Progress 
(NAEP). 

Goals of the Assessment 

National Assessment provides information to 
educational decisionmakers and practitioners 
that can be used to Identify educational 
problem areas, to establish educational prior* 
ities and to determine the national progress in 
education. To do so, NAEP must remain 
flexible:^^nough to accommodate possible 
extensions, refinements and modifications. 
The following goals have been established for 
the project by the National Assessment Policy 
Committee, the Analysis Advisory Commit- 
tee^ and the NAEP staff. 



Goal I: To measure change in the educa- 
tional attainments of young 
Americans. 

Goal II: To make available on a con- 
tinuing basis comprehensive data 
on the educational attainments 
of young Americans. 

Goallll: To utilize the capabilities of 
National Assessment to conduct 
special interest •*probes" into se- 
lected areas of educational 
attainment. 

Goal IV: To provide data, analyses and 
reports understandable to, inter- 
pretable by and responsive to the 
needs of a variety of audiences. 

Goal V: To encourage and facilitate inter- 
pretive studies of NAEP data, 
thereby generating implications 
useful to educational prac- 
titioners and decisionmakers. 



•Names of the Policy Committee and the Analysis 
Advisory Comtrlttee members are given in Appendix 
C and Appendix D, respectively. 



Goal VI: To facilitate the use of NAEP 
technology at state and local 
levels when appropriate. 

Goal VII: To continue to develop, test and 
refine the technologies necessary 
for gathering and analyzing 
NAEP achievement data. 

Goal 7III: To conduct an ongoing program 
of research and operational stud- 
ies necessar>^ for the resolution 
of problems and refinement of 
the NAEP model. (Implicit in 
this goal is the conduct of re- 
search to support previously 
mentioned goals.) 



Methodology 

To measure the nation's educational progress, 
National Assessment estimates the percentage 
of respondents (at four age levels) who are 
able to answer a question acceptably or 
perform a task. Each question or task (called 
an exercise) reflects a previously defined 
educational goal or objective. The exercises 
are administered to scientifically selected sam- 
ples (which take into account size of com* 
munity and socioeconomic status and include 
respondents from all 50 states). Students are 
sampled at three age levels that represent 
educational milestones attained by most stu* 
dents: age 9, when most students have been 
exposed to the basic program of primary 
education; age 13, when most students have 
finished their elementary school education; 
and age 17, when most students are still in 
school and completing their secondary educa- 
tion. To accurately reflect the skills, know- 
ledge and attitudes of the 17-year-olds, 
National Assessment also samples 17-year-olds 
not enrolled in school. Young adults (ages 28 
to 35) are assessed to determine the skills, 
knowledge and attitudes of those who have 
completed their formal education and have 
probably been aws^ from school for a num- 
ber of years. The samples are designed so that 
sound inferences can be made about the 
populations firom which the samples were 
selected. 
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NAEP does not develop or use scores for 
individual respondents. Rather, it determines 
how the four age levels perform on speeific 
exorcises and, within each age level, how 
groups of individuals (based on demographic 
and sociological variables) perform. Thus, it is 
not necessary for each respondent to take 
every exercise. The exercises are divided into 
booklets, and each in-school respondent takes 
only one booklet. Since the samples for the 
different booklets are statistically equivalent, 
group comparisons can be made across book- 
lets. This allows National Assessment to assess 
performance on far more exercises in a 
learning area than would be possible in the 
usual one-hour testing situation and provides 
broader coverage of the Assessment objectives 
for each learning area. 

While multiple-choice exercises pi'edominate, 
many open-ended exercises requiring any- 
where from a few words to a long essay as an 
answer are included in each assessment. Exer- 
cise writers are instructed to use the exercise 
format that provides the best and most direct 
measure of the objective being assessed. They 
are encouraged to develop exercises that 
employ the use of pictures, tapes, films or 
practical, everyday items as slimuli. Individual 
interviews, the manipulation of apparatus to 
solve a problem and observations of the 
respondents' problem-solving techniques arc 
used to supplement the usual paper-and-pencil 
tasks. For example, in music respondents 
were asked to sing a song or perform on an 
instrument; in science respondents were asked 
to conduct a small experiment; in mathe- 
matics respondents were asked to make 
change from a change drawer; in social studies 
respondents were asked to interpret an elec- 
tion ballot. 

NAEP regards positive attitudes toward or 
opinions about the various learning areas as 
important educational attainments. There- 
fore, affective exercises and attitude survey 
questions are also included in each assess- 
ment. 

Since individuals are not ranked according to 
their performance on the assessment materi- 



als, National Assessment does not emphasize 
the use of exercises with high discrimination 
power. The aim of the project is to describe 
attainment; this is best accomplished if the 
exercises used covev the entire spectrum of 
difficulty, from very easy tasks to the most 
difficult. 

Assessment exercises are administered either 
to individuals or to small groups (never larger 
than 12) by specially trained personnel.* 
Exercises specifically designed for individual 
administration include those having unusual 
stimuli or requiring something other than a 
written response. Some exercises at age 9 are 
administered individually so the youngest 
respondents can have the opportunity to 
express themselves verbally instead of by 
writing. In group administrations, all instruc- 
tions and the exercises themselves are present- 
ed to the respondents on paced tape record- 
ings prepared by professional tape script 
readers. This assures complete and uniform 
presentation of instructions and gives those 
who have a reading problem a chance to hear 
the exercise as they are reading it. 

To report the nation's educational progress, 
the project releases approximately one half of 
the exercises administered in a learning area. 
This allows the public to see the exercises and 
to evaluate the accompanying data in light of 
each exercise and its format. The other half of 
the exercises are kept confidential and are 
used to assess performance changes over time 
on specific exercises in the learning area. 

How does NAEP differ firom standardized 
achievement tests? Standardized achievement 
tests are norm referenced; National Assess- 
ment is content or objective referenced. With 
a standardized achievement test, each re- 
spondent takes every exercise, receives a score 
for his performance and is ranked on the basis 
of that score with respect to a reference 



Exercises are administered to all out-of-school re- 
spondents in a one-to one situation. About one 
fourth of the in-«chool respondents are assessed in a 
one-to-one situation. However, this proportion varies 
from age^evel to age level and from assessment year 
to assessment year. 
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grou|). With National Assessment no respond* 
ont takos all the exercises used to assess a 
learning area, no respondent receives a score 
and emphasis is t}laced on tlio performance of 
groups of respondents on specific exercises. A 
standardized achievement test is usually 
mass«admini8tered; NAEP exercises are admin- 
istered to small grout)s of no more than 12 
and also contain individually admiuistered 
sections. Standardized achievement test items 
are usually limited to a multiple*choice for* 
mat; National Assessment employs a wide 
variety of exercise formots. Standardized tests 
usually focus on the cognitive domain; NAEP 
includes exercises relating to the affective 
domain as well. A respondent is required to 
read the items himself when taking most 
standardized tests; Assessment exercises are 
read to respondents by a paced tape or the 
exercise administrator in an interview situa* 
tion except during the reading assessment. 
The items on a standardized achievement test 
are rarely, if ever, made publi';; the Assess* 
ment releases half of the exercises used in an 
assessment to accompany the data. 



The Assessment Cycle 

Much of the work of National Assessment is 
concerned with the year l)y year implementa- 
tion of m overall design and schedule which 
was developed at the inception of the project. 
Thus, many of the activities are baseline 
activities, repeated fur euch learning area and 
for each cycle in acrnrdance with an estab* 
lished design and %n ploying basically the 
same procedures and T#hnology. 

A single cycle of a learning area assessment, 
from objectives development or redevelop* 
ment to completion of the basic technical 
reporting of the data, requires approximately 
6 years. Development takes 3>/i years; V/t 
years are spent preparing for and performing 
data collection, and 1 year is required for 
preliminary analysis and basic reporting. 
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CHAPTER 2 
SAMPLING^ 



National Assessment's (NAEP) principal ob- 
jective is to assess the changes over time in 
educational achievement of four age groups in 
the United States. In addition to the national 
results for each age group, results are reported 
within each age level for certain major sub- 
populations, such as geographic region and 
sex. Current reporting populations and 
subpopulations are shown In Exhibit 24; the 
definitions for each are presented in 
Appendix A. 

EXHIBIT 2*1. National Assessment Reporting Groups 



Classification 
Age level 

Sex 

Geographic region 



Level of parental 
education 



Size and type of 
community (STOC) 



Color 



Subgroup 

9-year-olds 
13-year-olds 
1 7-year-olds 

Young adults (26-35 years) 

Male 
Female 

Northeast 
Southeast 
Central 
West 

No high school 
Some high school 
Graduated high school 
Post high school 

Low metropolitan 
High metropolitan 
Extreme rural 
Main big city 
Urban fringe 
Medium city 
Small places 

White 
Black 



There are two ways to meet this objective; 
conduct a census of all members of the four 
age populations or utilize probability samples 
of the four populations. A complete census 
has the advantages of providing great pre- 
cision and of allowing very detailed reporting. 
However, the four populations are very large 
(about 37 million people altogether); finding 
and assessing that many people would involve 
enormous expenditures of time and resources 
for daca collection, scoring and analysis. 
Furthermore, it would be extremely difficult 
to adequately train and supervise the large 
field staff necessary to assess all members of 
the population. 

Since National Assessment only collects data 
about a limited number of subpopulations 
and not individual students, schools, school 
districts or even* states, relatively small prob- 
ability samples allow estimation of educa* 
tional performance in these subpopulations 
with the required degree of precision. Fur- 
ther, by limiting the scale of data collection 
activities, it is possible to use a small, highly 
trained professional field staff to collect data 
of higher quality than would be possible with 
a census. Quantity of data is traded for much 
higher quality at reasonable cost. 



This chapter provides a brief overview of National 
Assessment sample designs for assessment Yean 
02-04. For more detailed documentation, see J.R. 
Chromy et ah, The National Assessment Approach to 
Sampling (Denver, Colorado: National Assessment of 
Educational Progress, 1974). For an introductory 
treatment aimed at state and local assessment prob- 
lems, see R.M. Jaeger^ A Primer on Sampling for 
Statewide Assessment (Princeton, N.J.: Educational 
Testing Service, Center for Statewide Educational 
Assessment, 1973). 
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The development of sample designs was based 
upon a long series of interactions between 
what the Assessment ought optimally to 
accomplish and what it could accomplish, 
given the available finite resources. Policy, 
data collection, analysis, reporting, cost and 
time considerations determined what types of 
sample designs were possible. The sample 
designs, in turn, put major constraints on the 
options available for data collection, analysis 
and reporting. For example, the limits placed 
on the number of administrations in a school 
and the size of group administrations (12 
students) are a compromise between cost 
efficiency and practical feasibility considera- 
tions* 



Design Specifications 

The target populations in the Year 03 and 04 
assessments included 9, 13 and 17-year-olds 
enrolled in public or private school, 17-year- 
oids who either left school before graduating 
or graduated early and young adults 26 to 35 
years old. Age-eligible persons who were 
non<*English speaking, institutionalized or 
handicapped (physically, mentally or emo- 
tionally) in such a way that they could not 
respond to the exercises as administered, were 
excluded. 

At each age level the sample for each package 
of exercises was designed to meet the follow** 
ing specifications: 

1. Adequate representation of the sub- 
populations to allow estimation of the 
desired proportions with an acceptable 
level of precision. 

2. Representation by at least one sample 
point for each of the states and the 
District of Columbia. The design was 
not, however, to provide for making 
comparisons among states, school dis- 
tricts, schools, teachers or individual 
students. 

3. Facilitation of field-operating proce- 
dures. 



NAEP's policy that administration of mate* 
rials was not to take more than one class 
period of a student's time, that the uomands 
on school personnel were to be minimized 
(one package per student) and that the 
number of students assessed m any one school 
was to be limited to 12 also placed constraints 
on the sample d« sign. Among other important 
sampling considerations were (1) that no 
more than about one half of the group- 
administered packages were to be admin- 
istered in any one school, (2) that out-of- 
school 17-year-olds and young adults were to 
be allowed to take up to four packages and 
(3) that the respondents taking each package 
were to be a probability subsample of the 
total sample for the age group. 

This design provided for simple, precise esti- 
mates of population proportions. Since results 
were to be reported by individual exercise, 
these estimates were to be relatively inexpen- 
sive to compute. 

To meet these specifications, two three-stage, 
deeply stratified cluster designs with extra 
sampling of certain strata were developed. 
One design was for the survey of 9, 13 and 
17-year-olds enrolled in school. The out-of- 
school design was developed to assess young 
adults aged 26 to 35 and 17 -year-olds not 
enrolled in school. 



In-Schuol Sample Design 
Stratification 

In a stratified design the population is divided 
into two or more groups, or strata. Samples 
are then drawn from each stratum rather than 
from the population as a whole. The two 
major reasons for stratification are to insure 
representation of specified subpopulations 
and to achieve more precise estimates. Nation- 
al Assessment samples were stratified by 
geographic region, state, size of community 
and socioeconomic level as shown in Exhibit 
2-2. Socioecono mic stratification was 
performed at the school level in the two 
largest size of community (SOC) strata and at 
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the primary sampling unit level in the remain- 
ing two size of community strata. 

EXHIBIT 2-2. National Assessment Sampling Strata 
Classification Strata 



Region 



Size of 

community (SOC) 



Socioeconomic 
level 



Northeast 
Southeast 
Central 
West 

Counties containing cities with 
1 970 populations of 1 80,000 or 
greater 

Counties in the same Standard 
Metropolitan Statistical Area 
(SMSA) as the above counties 

Counties not in the above 
strata that are part of an SMSA 
or contain a city with a 1 970 
population of 25»000 or greater 

All other counties 

High -schools in the first two 
SOC strata 

Low-counties in the last two 
SOC strata 



Multi-stage Sampling 

A multi-stage design involves sampling in 
successive steps or stages in order to control 
sampling and date collection costs. 

In National Assessment's in-school sample 
design, the first stages— or primary sampling 
units (PSUs)— consisted c' 'ounties or groups 
of contiguous counties. A sampling frame of 
PSUs was constructed with United States 
Census data on the number of persons by age 
in each PSU. The PSUs were stratified by 
region, and within region by state, size of 
community and, for the two smaller size of 
community strata, by socioeconomic level. 
From this stratified list of PSUs, a probability 
sample of 116 PSUs was independently drawn 
each year. 

At the second stage, a list was made of all 
public and private schools in each selected 
PSU. The listing included the estimated num- 



ber of 9, 13 and 17-year-olds enrolled in each 
school. A secondary sample of schools was 
then drawn for each sample PSU. Schools in 
the two larger size of community strata were 
stratified by socioeconomic level prior to 
selecting the secondary sample. 

In each selected school, the third-stage sam- 
pling units consisted of eligible students en- 
rolled in the school. Every eligible student 
was listed. A random sample of students was 
then drawn and randomly assigned one of the 
assessment packages scheduled for that 
school. 

To ensure adequate sample sizes for each of 
the various reporting groups listed in Exhibit 
2-1, it was necessary to increase the sample 
size of the population at the lower end of the 
socioeconomic scale. This "oversampling** was 
performed at the first stage in the two smaller 
size of community strata and at the second 
stage in the two remaining size of community 
strata. 



Out-of-School Sample Design 

The out-of-school sample design is similar to 
the in-school design. As with the in-school 
design, primary sampling units consisted of 
counties or groups of contiguous counties 
stratified by region, state, size of community 
and socioeconomic level. One hundred four 
primary sampling imits were drawn and used 
for both Year 03 and Year 04 (one half of 
them were used in Year 02). Each year, a new 
secondary sample of small land areas (seg- 
ments) was drawn in each PSU. All of the 
housing units in each segment were listed and 
a sample of the housing units was screened for 
eligible out-of-school 17-year-olds and adults. 



The Supplementary Frame 

In order to increase the sample size of 
17-year-olds not enrolled in school, a proba- 
bility subsample of one half of the schools in 
the 17-year-old assessment was drawn. Each 
school was asked to provide lists of persons 
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who had either graduated early or dropped 
out of school. A subsample of dropouts and 
early graduates was drawn. 

Sample Sizes 

The size of a sample required for estimating 
baseline performance and change over time is 
related to the type of administration, the 
minimum change to be detected with a given 
degree of confidence, the desired power, the 
sample design, operational procedures, time 
and cost. In accounting for these, NAEP set 
the planned sample size for the in-school 
samples at 12 individuals per group- 
administration session and the planned 
national sample sizes at 2,592 for group- 
administered packages and 2,160 for individ- 
ually administered packages. 

One problem recognized in planning the 
sample sizes for the in-school assessment was 
the reduction in sample size due to absen- 
teeism at the time of assessment. Therefore, 
to assure that the desired sample sizes were 
achieved, a random sample of 16 students was 
selected for each group administration. The 
first 12 students were assigned to the assess- 
ment session. The remaining 4, designated as 



alternates, were used to replace any of the 12 
students v/ho were absent at the time sched- 
uled for the package administration. For each 
individual administration, 2 students were 
randomly selected with the second serving as 
an alternate for the first. Special selection 
procedures were adopted for the in-school 
sample to accommodate schools enrolling less 
than the required number of students for one 
group administration. The allocations to those 
schools were in addition to the planned 
national sample sizes given above. 



Response Experience 

The response experiences for Year 03 and 
Year 04 are summarized in Exhibits 2-3 
through 2-6. The observed sample sizes per 
package are presented in Exhibits 2-3 and 2-4. 
Exhibits 2-5 and 2*6 present the average size 
per package for each of the reporting groups. 
The planned sample sizes were 2,592 respond- 
ents per package for in-school group admin- 
istrations, 2,160 per package for in-school 
individualized administrations and 2,000 re- 
spondents for each package in the out-of- 
school sample. The exhibits reveal that the 
observed sample sizes were very close to the 
projected sizes. 
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EXHIBIT 2-3. Year 03 (Music and 
Social Studies) Observed Sample Size 



Package Number Age Level 





o 


t X 
1 9 


1 / 


AdUlt 


1 


2,717 


2,661 


2,684 


2,353 


2 








2,328 


3 


2,733 


2,600 


2,654 


2,350 


4 


2.661 


2,664 


2,707 


2,335 


5 


2,296* 


2,667 




2,367 


6 


2.321 


2,741 


2,659 


2,334 


7 


2,341 




2,665 


2,356 


8 




2,286* 


2,670 


2,337 


9 




2,297 


2.722 




10 




2,287 


2,498* 




11 






2,498 




12 






2,497 




Packages administered: 


17.781 


25,569 


31.605 


18,760 


Respondents: 


17,781 


25,569 


28,812 


4,822 


Total number of packages administered at all 


ages: 




93,715 


Total number of respondents 


at all ages: 




76,984t 



*Sample sizes below line are for individually administered packages. 

f Adults and out-of-school 1 J-year-olds were allowed to take more than one package. 



EXHIBIT 2-4. Year 04 (Science and 
Mathematics) Observed Sample Size 

Package Number Age Level 



1 

2 

3 

4 

S 

6 

7 

8 

9 
10 
11 
12 
13 
14 

Packages administered: 
Respondents: 

Total number of packages administered at all ages: 108,850 
Total number of respondents at all ages: 93i438i' 

*Sample sizes below line are for individually administered packages. 

f Adults and out'of'school J 7-year-olds were allowed to take more than one package. 



9 


13 


17 


Adult 


2,666 


2,531 


2,651 


1,993 


2,670 


2,648 


2,713 


1,987 


2,628 


2,674 


2,756 


1,999 


2,646 


2,621 


2,721 


2,004 


2,670 


2,684 


2,751 


2,030 


2,674 


2,512 


2,725 


2,036 


2.684 


■ 2,689 


2,672 


2,034 


2,258* 


2,575 


2,682 


2,025 


2,283 


2.573 


2,638 


2,224 


2,269> 


2,633 






2,254 


2.643 






2,217 


2,507* 








2,502 








2,498 




25,403 


30,247 


37,092 


16,108 


25,403 


30,247 


33,577 


4,211 
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EXHIBIT 2-5. Average Number of Responses by Group and Type of Administration 
at Each Age Level-Year 03 (Music and Social Studies) 



Age Level 



13 



17 



Adult 





Group 


inoividuat 


Group 


inaiviau&i 




irQlYiQUal 




National 


2,706 


2,307 


2,672 


2,290 


2,679 


2,498 


2,345 


Region 












601 


519 


Northeast 


674 


■ • 574 


651 


558 


644 


Southeast 


696 


593 


679 


582 


702 


654 


582 


Central 


668 


570 


681 


583 


667 


622 


632 


West 


669 


570 


661 


567 


665 


621 


612 


Sex 
















Male 


1.338 


1,140 


1,337 


1,146 


1,296 


1,209 


1,060 


Female 


1,368 


1,167 


1,335 


1,144 


1,383 


1,289 


1,285 


Color 














1,968 


White 


2,106 


1,796 


2,138 


1,832 


2,177 


2,030 


Black 


414 


353 


352 


301 


335 


313 


240 


Other 


186 


158 


183 


157 


167 


ISS 


137 



Parental education 
No high school 
Some high school 
Graduated high school 
Post high school 
Unknown 

Size and type of community (STOC) 
Low metro 
Extreme rural 
Small place 
Medium city 
Main big city 
Urban fringe 
High metro 



157 


134 


196 


168 


266 


248 


705 


144 


122 


261 


223 


373 


348 


394 


647 


552 


825 


707 


816 


761 


662 


896 


764 


1,025 


878 


1,074 


1,002 


499 


862 


735 


365 


313 


149 


139 


84 



277 
270 
877 
454 
283 
271 
272 



236 
231 
748 
387 
242 
231 
232 



264 
255 
899 
443 
299 
225 
287 



227 
219 
771 
379 
256 
193 
246 



312 
310 
886 
438 
240 
144 
349 



291 
289 
826 
408 
224 
135 
325 



233 

238 

746 

32 

188 

384 

234 



1ft 
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EXHIBIT 2-6. Average Number of Responses by Group and Type of Administration 
at Each Age Level-Year 04 (Science and Mathematics) 



Age Level 

9 13 17 Adult 



Group Individual Group Individual Group Individual 



V t A. * t 

National 


2,663 


2.255 


2,612 


2,247 


2,690 


2,502 


2.014 


Region 
















Northeast 


656 


5ol 


65 1 


565 


653 


620 


477 


Southeast 


669 


565 


667 


565 


68S 


641 


O wO 


Central 


672 


566 


649 


562 


679 


618 


516 


West 


665 


563 


645 


555 


673 


623 


512 


Sex 
















Male 


1,328 


1,123 


1,294 


1,098 


1,280 


1,206 


925 


Female 


1,335 


1,132 


1,318 


1,148 


1,409 


1,296 


1.088 


Color 
















White 


1,997 


1,689 


1,977 


1,683 


2,1 10 


1,942 


1.688 


Black 


466 


403 


436 


391 


416 


413 


200 


Other 


199 


163 


199 


173 


163 


147 


126 


Parental education 
















No high school 


146 


114 


185 


185 


248 


223 


562 


Some high school 


125 


120 


232 


230 


353 


335 


325 


Graduated high school 


564 


566 


792 


760 


816 


782 


583 


Post high school 


787 


819 


994 


837 


1,108 


1,025 


481 


Unknown 


1,041 


637 


409 


234 


164 


137 


62 


Size and type of community (STOC) 
















Low metro 


265 


229 


263 


222 


274 


280 


200 


Extreme rural 


266 


227 


264 


221 


270 


238 


197 


Small place 


891 


743 


869 


740 


966 


876 


847 


Medium city 


372 


326 


365 


308 


380 


357 


232 


Main big city 


285 


247 


277 


251 


214 


204 


140 


Urban fringe 


317 


261 


314 


275 


321 


300 


199 


High metro 


267 


222 


260 


229 


265 


247 


199 






1^ 
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CHAPTER 3 
OBJECTIVES AND EXERCISE DEVELOPMENT 



The development of objectives and exercises 
occurs in five steps: 

1. development and review of objectives;* 

2. exercise preparation and local try outs; 

3. exercise review and revision; 

4. field testing, exercise scoring and review 
of the results; and 

5. final exercise review and selection. 



Development and Review of Objectives 

Whether objectives were developed for new 
areas (math, music, social studies) or redevel- 
oped for previously assessed areas (science) 
the final version had to satisfy a cross-section 
of subject-matter specialists and educators in 
each learning area. Acceptance and approval 
of each set of objectives as an important set 
of educational goals was also elicited from 
concerned citizens. Whenever National Assess- 
ment (NAEP) has employed consultants for 
review conferences and approval of materials, 
attempts have been made to insure representa- 
tion from all regions of the country, participa- 
tion of members of minority groups and 
participation of both males and females. 

In addition, the Assessment selected subject* 
matter specialists and educators representing 
different specialties within the discipline; dif- 
ferent organizations and projects associated 



Objectives development and review in specinc learn- 
ing areas are described in the objectives booklets for 
those areas. 



with the learning area or discipline; and both 
elementary and secondary levels of public, 
private and parochial school people (class- 
room teachers, curriculum specialists, admin- 
istrators). 

Further selection requirements for concerned 
citizens included representation from differ- 
ent types of communities, diverse national 
organizations and different occupations. 

Objectives development took approximately 
one year to accomplish for each learning area. 
Contractors and consultants conducted a liter- 
ature search within each area to document 
curriculum trends, existing sets of educational 
objectives and general content organizations. 
Then subject-matter specialists participated in 
a series of development and review confer- 
ences which yielded a first draft of objectives. 
Further reviews, including a review by know- 
ledgeable, concerned citizens, were conduct- 
ed. Once the objectives were revised and 
approved, the final draft was again reviewed 
and approved by another group of subject- 
matter experts. The new concensus objectives 
were adopted by National Assessment. 



Exercise Preparation and Local Tryouts 

Based on the objectives for each learning area, 
NAEP developed specifications for the num- 
ber, character and quality of exercises to be 
developed. Special emphasis was given to 
writer qualifications, documentation, diffi- 
culty levels and usefulness at more than one 
age level. Exercise writers had to be subject- 
matter experts with experience in the educa- 
tion of 9-year-olds, 13-year-okis, 17-year-olds 
or adults in the particular learning area. 



2Q 



BEST copy mmu 



Exercise documentation had to include a 
rationale; administration, scoring and report- 
ing directions; scoring keys or scoring cate* 
gories; sample responses and all special stimuli 
(e.g., music and the related permissions for 
use). 

Exercises had to be developed with difficulty 
levels ranging from an expected response rate 
of over 75% correct responses to an expected 
rate of less than 25% correct responses. This 
requirement was made because National 
Assessment must be able to describe a broad 
range of educational attainments achieved by 
the groups of people in the target popula- 
tions. 

Development of exercises that could be 
administered at two or more age levels was 
encouraged because of the useful comparisons 
that could be made between ages. 

As pait of the exercise preparation process, 
exercise developers conducted small-scale 
local tryouts for each exercise they submitted 
to the project. These tryouts served two 
functions: 

1. They furnished information about exer- 
cise clarity and administrative feasi- 
bility. 

2. They provided sample responses for 
open-ended exercises which the con- 
tractor used to develop initial scoring 
guides. 

Exercise Review and Revision 

Newly developed exercises were reviewed by 
National Assessment staff and the Exercise 
Development Advisory Group (EDAG), which 
has a rotating membership comprised of five 
educational measurement specialists. Subject- 
matter specialists critiqued each exercise with 
respect to whether it was a direct, clear 
measure of an objective for the learning area, 
whether it was relevant to current educational 
practices and opinions and whether it contain- 
ed accurate scoring and reporting guides. 

2|( 



Groups of informed and concerned citizens 
reviewed each exercise with respect to wheth- 
er it was an appropriate measure of an 
objective for a given age, whether it might be 
offensive to any particular group of people 
and whether it was a relevant exercise in 
terms of what they considered to be valuable 
learning experiences. Criticisms and sugges- 
tions were transmitted to the exercise devel- 
opment contractor, who then reviewed all the 
data, suggestions and criticisms and revised 
the exercises accordingly. 



Field Testing, Exercise Scoring 
and Review of the Results 

Following review and revision, exercises were 
given a national tryout. Tryout respondents 
were selected to give representation to the 
reporting categories: region of the country, 
size and type of community, socioeconomic 
levels, color and sex. The tryouts served three 
purposes: 

1. to check administrative feasibilities, 

2. to provide data for improvement of 
scoring guides and 

3. to provide data necessary for evaluating 
exercises in order to select those that 
would be used in the assessment. 

Responses were scored separately for each 
age. Multiple-choice exercise responses were 
tallied directly by scoring clerks. Short-answer 
written responses were categorized according 
to scoring guide categories, and the number of 
responses falling in each category was tallied. 
Recording response data for exercises that 
involved longer open*ended responses, per- 
formance or production was a more complex 
process. Subject-matter specialists scored 
these exercises using the following general 
pattern: 

1. reviewing all responses to a given 
exercise, 

2. outlining response categories that were 
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both relevant to the exercise objective 
and inclusive of the range of responses, 

3, tallying responses occurring in each ten- 
tative category, 

4, reviewing and revising the categories and 

5, tallying responses for the final cate- 
gories. 

Subject-matter specialists carefully reviewed 
all response data from all exercises. Each 
scoring guide or exercise was checked for 
subject-matter accuracy, appropriateness for 
the age level and biases with respect to any 
particular group of people. These exercises 
were also reviewed by NAEP staff members 
and EDAG. They considered each exercise in 
light of potential administration, scoring and 



reporting difficulties, as well as general meas- 
urement characteristics. These considerations, 
taken together with the subject-matter spe- 
cialists' recommendations, determined the 
exercises suitable for use in the assessment. 



Final Exercise Review and Selection 

The pool of suitable exercises had to be 
narrowed to the number that could actually 
be used in the assessment. Therefore, the 
exercises were again reviewed by subject- 
matter specialists and selected on the basis of 
exercise quality, importance, difficulty and 
coverage of desired objectives. This final set 
of exercises was then forwarded to the United 
States Office of Education for review and 
approval. 



2% 
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CHAPTER 4 

PREPARATION OF MATERIALS AND ADMINISTRATION OF THE ASSESSMENT 



Before the data collection for an assessment 
year begins, materials are prepared for use in 
the field. Exercises which have been selected 
for inclusion in the assessment are grouped 
into booklets-called "packages"~and print- 
ed. Tape recordings of the exercises are 
produced to accompany the packages. Exer- 
cise administrators are trained, and materials 
are shipped to them. All of these activities are 
planned, coordinated and monitored by 
National Assessment (NAEP) staff. 

In preparation for the music and social studies 
assessments (1971-72) and the science and 
math assessments (1972—73), the Research 
Triangle Institute (RTI) of Raleigh, North 
Carolina, assembled the exercise packages and 
produced the tape recording. The Measure- 
ment Research Center (MRC) of Iowa City, 
Iowa, printed the packages so that the re- 
sponses, which were marked on the package 
pages, could be read and recorded by 
machines. RTI, with a subcontract to MRC 
for the Central and Western regions, conduct- 
ed the field administration, and MRC ulti- 
mately conducted the scoring and machine 
processing of the exercise booklets. 



Preparation of Materials 

Since individuals of different ages were to 
receive somewhat different sets of exercises, 
packaging was done separately for each age 
level in the assessment. Packages were com- 
piled according to the following criteria: 

1. Each package contained 35 minutes of 
exercise time. An additional 15 minutes 
of administration time was allowed for 
introductory statements, instructions 



and completion of personal background 
questions. For in-school administrations 
no more than about 60 minutes of a 
student's time was required for partici- 
pation in an assessment since each stu- 
dent was allowed to take only one 
package of materials. Therefore, in most 
schools a package administration could 
be completed within the time bound- 
aries of a class period. 

2. Each package contained some exercises 
from both learning areas being assessed 
that year. 

3. Each package contained exercises from 
the three levels of difficulty (easy, 
medium, difficult). 

4. Exercises could not appear in the same 
package if they used the same or similar 
stimulus materials or if one exercise 
provided a clue to the answer of 
another. 

5. Each package contained exercises cover- 
ing several topics in each learning area. 

Some exercises were designed to be admin- 
istered to one individual at a time in an 
interview situation. Most exercises, however, 
were designed to be completed by respond- 
ents on their own as they read the questions 
and heard them read on tape. The respond- 
ents either marked an oval beside a multiple- 
choice answer or wrote open-ended responses 
on lines provided in the booklet. Since 
respondents worked these exercises on their 
own, up to 12 students could be assessed as a 
group in one administration. In each assess- 
ment year there were three packages of 
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individual exercises for each a?e level and 
from four to nine packages of group exercises. 

The package printing procedures add'^d coded 
and machine-readable identifying information 
to each exercise page. Quality-control meas* 
ures insured that all machine-readable infor- 
mation was accurately located on the pages 
and that paper and i!ik quality met standards 
necessary for later machine processing of the 
exercise booklets. 

To assure uniformity in administration, and 
to help remove biases as well as aid respond- 
ents with reading difficulties, tape recordings 
were produced to accompany each group 
package. Exercise directions, written stimulus 
materials, exercise stems and exercise foils 
were presented both on the tape and in the 
package. Some exercises in the math assess- 
ment were an exception to this rule because it 
would have been inappropriate to have had 
certain arithmetic symbols read aloud on 
tape. The capes were paced to allow an 
adequate amount of response time for each 
exercise. Individual packages generally did not 
have accompanying tapes, since the adminis- 
trator read tlie questions in an interview 
situation. 

Some exercises required the preparation of 
additional stimulus materials. Several music 
exercises required that musical stimuli be 
presented on tapes. Music performance exer- 
cises also required that the administrator 
record responses on a second tape recorder. 
Individual exercises in other learning areas 
sometimes involved the use of other materials 
or equipment, such as pictures, rulers and 
science apparatus, which were purchased or 
made especially for use in an assessment. 

After all of the materials for the assessment of 
an agi» group were prepared, the field staff 
was instructed about details on how to 
administer the packages and procedures for 
contacting schools or locating and sampling 
individuals for the out-of-schcol assessment. 
The assessment year schedule for administra* 
tion was as follows: 13-year-olds were as- 
sessed in the fall (mid-September through 



mid -December); 9-year-olds were assessed in 
January and February; and in-school 17-year- 
olds were assessed in March and April. The 
assessment of adults and out-of-school 17* 
year-olds occurred in the spring and early 
summer months. 



Administration of the Assessment 

Some aspects of assessment administration 
procedures varied depending on whether the 
respondents were m school or out of school. 
The sampling methods used by the two field 
staffs were not the same, and the procedures 
for administration, such as the number of 
packages administered and the use of tape 
recordings, also differed. 



in*school Administration: 9, 13 and 17-YearOids 

Before the packages could be administered in 
the selected schools, cooperation had to be 
obtained from school personnel, and opera- 
tional procedures had to be established 
between them and the administration con- 
tractor's field staff. Chief state school officers 
were informed of all schools selected for the 
assessment within their respective states. The 
National Assessment staff director notified 
superintendents and private school officials 
that schools from their districts had been 
selected for participation. The superintend- 
ents also received materials from the sampling 
and administration contractor which identi- 
fied the selected schools in their districts, 
described the project and suggested dates for 
meetings with members of the field staff. 

The field staff included 4 regional supervisors 
and 29 district supervisors. Each district 
supervisor met with the school officials in his 
district to explain the purpose of NAEP, 
describe the operational procedures for 
completing the assessment and determine a 
time suitable for assessment in the area. The 
school officials were also asked to respond to 
principalis questionnaire items. Questions 
were asked about the school's enrollment in 
various grades, the types of communities in 
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which the studonts lived, the general occupa- 
tional levels of the parents in the community 
and, in the case of the science assessment, the 
type of science curriculum used by the 
school. 

Each school principal appointed a coordinator 
who arranged for space to conduct the group 
and individual administrations and who 
worked with the district supervisor to arrange 
a mutually convenient schedule in the school 
and to ensure that students arrived on time 
for their scheduled administrations. The coor- 
dinator also arranged to provide a listing of 
each student bom during the birth-date range 
defining National Assessment eligibility. The 
district supervisor used the listing to make a 
random selection of the students to be assess- 
ed; each student was assigned to receive one 
assessment package. 

After the sample was selected in a school, 
package administrations were done by the 
district supervisor or by an exercise adminis- 
trator hired locally and trained by the district 
supervisor. The exercise administrators had 
various backgrounds including teaching, 
substitute teaching and survey research. 
Assessment time varied between schools de- 
pending upon the number of packages assign- 
ed to each school; only rarely did administra- 
tions within a school take more than three 
days. 

The administrator coded each student's birth 
date, sex, grade, color and identification 
number on his or her package. The district 
supervisor checked all data coded on the 
packages against the student listing and in- 
structed the school coordinator to save the 
listing for six months in case it might be 
needed for data verification. Since names are 
not associated with NAEP packages, the 
listing which cross-referenced packages by 
identification numbers was the only means of 
verifying lost or questionable data. After the 
six-month period, the listing was destroyed to 
protect the anonymity of students who parti- 
cipated in the assessment. 



The district supervisor sent the completed 
packages to the scoring contractor where they 
were quality checked to verify that correct 
administrative procedures were being fol- 
lowed by the field staff and that all packages 
were accounted for. The coded identification 
information was also checked for accuracy; 
inconsistencies that could not be reconciled 
were sent back to the district supervisor to be 
checked against the student listing. 

Each school that participated in the assess- 
ment received a thank-you letter and a ques- 
tionnaire concerning contact procedures, 
preparation for the assessment, assessment 
exercises, personal data questions and the 
school's general involvement during assess- 
ment week. This information was analyzed to 
discover problems in the field procedures so 
that they could be corrected in future 
assessments. 



Out-of-School Administration: 17- Year-Olds 

In order to insure that 17-year-old results 
would be representative of the entire popula- 
tion of 17-year-olds, efforts were made to 
locate and assess those 17-year-olds not en- 
rolled in school during the time of the 
in-school assessment. These 17-year-olds 
either had withdrawn from school or com- 
pleted requirements for early graduation. 

There were two methods of identifying and 
locating ehgible out-of-school 17-year-oIds. 
First, during their visits to schools for the 
in-schooI administrations, the district super- 
visors obtained lists of out-of-school 17.year- 
olds. A sample of potential respondents was 
drawn from these lists and, during the early 
summer months, the district supervisors 
attempted to locate and assess these individ- 
uals. Second, if the field staff in charge of the 
adult assessment located any age-eligible 
17-year-oIds who were not in school, they 
administered 17-year-old packages to these 
individuals. 
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Each year approximately 1»00() out-of-school 
17-year-olds were located and assessed. The 
administration procedures used were the same 
as those used for the adults. 



OutofSchooi Administration: Young Adults 

Locating younfi adults ages 26-^^35 was more 
difficult than locating students, and a sepa- 
rate, specially trained field staff was assigned 
to conduct the adult assessment. They used 
different sampling techniques for selecting 
respondents and their administrative proce- 
dures were different from those of the in- 
school administration staff. 

The field supervisors first identified and 
visited the areas of the country included in 
the sample. Next, they compiled a list of all 
the sample housing units that fell within the 
area boundaries. The field interviewers then 
canvassed the sample housing units to find 
age-eligible adults as well as out-of-school 
17-year-olds mentioned previously. The inter- 
viewers filled out a screening questionnaire on 
which they listed all household members and 
their birth dates. The interview was termi- 
nated if the screening questionnaire listed no 
age-eligible household members. The inter- 
viewers made call-backs to households where 
no one was at home, thereby completing 
screening questionnaires on more than 99% of 
the sample housing units. 

When an eligible individual was located, the 
field interviewer explained the amount of 
involvement that would be required and 
encouraged participation in the assessment. 
Handout materials were available to anyone 
who desired more information about the 
project. Eligible individuals who refused to 
participate were later contacted by the field 
supervisor, and after receiving more informa- 
tion they often agreed to participate. Of those 



who were contacted in each assessment year 
approximately 84% of the eligible young 
adults and 91% of the out-of-school 17-year- 
olds participated in the assessment. 

The one-hour time restriction was not appHed 
to out-of-school administrations. Eligible 
young adults and out-of-school 17-year-olds 
were allowed to take as many as four pack- 
ages of materials. Almost all of the respond- 
ents chose to take four packages. They 
received $5.00 for each package they com- 
pleted. Package assignment procedures 
insured that all packages were administered to 
about the same number of individuals and 
that each package was administered as the 
first, second, third or fourth package about 
the same number of times. Each individual 
also responded to items on a background 
questionnaire. 

Since the out-of-school assessment took place 
in the respondents' homes, the administration 
was always a one-to-one (interviewer and 
respondent) situation. No packages were 
administered in a group setting. For some 
exercises the interviewer asked the questions 
and wrote down or tape-recorded the individ- 
ual's responses. Other exercises were. **self- 
administerable'' and the respondent read the 
question while listening to it being read on 
tape. The tapes were not paced as in the 
in-school administrations. Instead, the field 
interviewer started and stopped the tape as 
necessary to allow the individual a reasonable 
length of time to respond. 

The field interviewers coded the individual's 
identification information on the packages, 
and the completed materials were sent to the 
scoring contractor where quality-control pro- 
cedures insured that all information was 
coded correctly and that all of the materials 
that had been shipped to the field had been 
returned. 
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CHAPTER 5 

CATEGORIZING RESPONSES AND PREPARING THE DATA FILES 



The data that National Assessment (NAEP) 
collects in the field must be coded into a form 
suitable for computer processing and analysis. 
Measurement Research Center (MRC) of Iowa 
City, Iowa, was the contractor in charge of 
receiving, categorizing and machine processing 
the data. Responses to open-ended and per* 
formance exercises were categorized and 
coded into machine-readable form by a 
special staff of scorers. Responses to multi* 
pie-choice exercises were read directly by 
optical scanning machines. These machines 
transfen'ed the information to computer data 
tapes which provided the basis for data 
analyses performed by the NAEP staff. 



Receipt Control 

When booklets were printed and shipped to 
the field staff, each one-week unit of v/ork 
was packed in a fiberboard box. Once the 
packages were administered, they were re- 
packed in the same box and returned to MRC 
for processing and scoring. From the begin- 
ning to the end of the MRC processing, that 
unit of work was kept intact. 

Upon receipt from the field, each box was 
unpacked by the receipt control staff and 
checked to ensure that: 

1. All forms and worksheets were properly 
and accurately completed. 

2. All materials (packages, tapes, work 
samples, etc.) were properly and unique- 
ly identified. 

3. All materials were accounted for. 




4. Personal data and documentation for 
each respondent were complete and 
consistent. 

5. Machine-readable responses were 
gridded according to instructions and 
were dark enough to be scorable. 

6. Administrator-recorded responses were 
legible and scorable. 

7. Branching exeicises were being correctly 
administered. 

MRC corrected errors and inconsistencies 
which could be solved with the information at 
hand. Problems for which there was insuf- 
ficient information were referred to the field 
staff for solution. In all cases, NAEP staff was 
notified of the problems and their recom- 
mended solutions. After the receipt control 
checks were completed, each box was repack- 
ed and given to the MRC hand-scoring staff. 



Ongoing Hand Scoring 

In each assessment year, MRC employed a 
sp'^cial staff to hand score open-ended exer- 
cises. Scorers were responsible for categoriz- 
ing responses according to predetermined 
guidelines and coding the category of the 
response into ovals that could be read by the 
optical scanning machine. 

Hand-scoring tasks ranged from simple tallies 
to rather high-level judgments. Hand scorers 
were required to have some academic back- 
ground and, preferably, teaching experience 
in the subject they were scoring. 



BEST COPY AVAIUBLE 



When the initial data (age 13) began to arrive 
from the field, the scoring staff was assembled 
for training. A training session took place 
prior to the scoring of each age level. The first 
training session was the most complex, since 
scorers had to become familiar with NAEP, 
the objectives related to the particular learn- 
ing area and general scoring and coding 
procedures. 

The MRC director of scoring conducted the 
training sessions in cooperation with NAEP 
staff. Once the general orientation was accom- 
plished, training began on specific exercises 
and their scoring guides. The guides were 
initially developed by the exercise developer, 
reviewed and refined by subject-matter and 
lay consultants and refined further by NAEP 
staff after tryouts. Acceptable and unaccept- 
able responses were defined through prose 
descriptions and sample responses. Response 
categories that were of interest to NAEP staff 
and consultants were also defined. 

Scorers were trained in the use of the guides 
by scoring sample responses, which were 
either taken from tryout material or the 
arriving assessment data. Scorers initially 
worked as a group and discussed the appro- 
priate categorization of each example re- 
sponse. They then worked individually on 
another set of responses. Discrepancies were 
resolved and explained. Once the group felt 
comfortable using the guides, they started 
categorizing the actual data. The director of 
hand scoring checked the scoring for the first 
20—24 papers for each exercise for each 
scorer. Incorrect categorizations were explain- 
ed and careless errors were pointed out. 
Subsequently, spot checks were made on the 
work of each scorer and retraining was con- 
ducted as needed. 

Based on the final data some scoring guides 
needed modification. Generally, this involved 
some redefinition of scoring categories. All 
responses scored before a scoring guide 
change were, of course, rescored. 

Scorers were approved by NAEP prior to 
training and could be removed at any time if 



their work was found unsatisfactory. Quality- 
assurance procedures were developed both by 
MRC and NAEP to ensure the consistency of 
hand scoring. Members of the NAEP staff 
made several trips to MRC to observe and 
assist in training sessions and to check that 
the hand scoring was consistent with the 
scoring guides furnished by the project. 

After all the data at a given age level was 
scored, MRC prepared a report which includ- 
ed the final form of the scoring guides, 
examples of responses for scoring guide cate- 
gories and a summary of the problems en- 
countered and their solutions. 



Programming and Machine Processing 

Once the open-ended responses were catego- 
rized, the packages were repacked and sent on 
for machine processing. While the exercises 
were undergoing hand scoring, all of the 
procedures and programs to be used in the 
machine processing were being tested on 
materials developed by MRC and NAEP staff. 
The optical scanning device was programmed 
to read the response data on the exercise 
pages and produce data tapes which could be 
used by computers. 

Once the packages arrived at the MRC proc- 
essing department, the pages were separated 
so they could be passed through the scanning 
device. Each page contained preprinted, 
machine-readable information such as age, 
package, exercise and page number. On pages 
with multiple-choice exercises the respond- 
ents had made pencil marks in ovals next to 
the answers they chose. For other exercises 
the hand-scoring staff had marked codings in 
preprinted oval positions on the page. The 
scanning device, through the use of reflected 
light, distinguished the position of each pen- 
cil-marked oval. The scanner assigned numer- 
ical values, specified in computer programs, to 
the various marks according to their position 
on the page. The coded information was 
transferred onto magnetic tapes which later 
were used to construct the NAEP respondent 
data files. 
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MRC's computer checked and edited the 
contents of the magnetic tapes for package 
counts, internal exercise consistency and 
response ranges before they were sent to 
National Assessment. Tapejj were also pro* 
vided to RTI, the sampling contractor, for 
calculating weights.* After scanning and edit- 
ing, exercises which required special scoring 
were isolated. The remaining pages were then 
sent to a climate-controlled warehouse for 
permanent storage. It is a policy of NAEP to 
save all source documents. 



Special Scoring 

Occasionally, the scoring of an exercise 
requires special scorer qualifications and 
training. 

Music exercises involving the evaluation of 
singing and playing an instrument, a mathe- 
matics exercise about computer programming 
and several social studies and science exercises 
were scored professionally by subject-matter 
experts. The mathematics exercise was scored 
by bona fide computer programmers who 
read only that one exercise. Professional 
scorers were trained to score all of the musical 



For a discus8ion of weighting procedures, see Chap- 
ter 7, Computation of P- Values. 



performance exercises. Judgments wore made 
based on the criteria established in scoring 
guides. For example, music scorers listened to 
tape recordings of respondents singing or 
playing instruments and then scored the 
renditions for accuracy of pitch and rhythm 
according to specified criteria. 

Special scoring often involved separate sched- 
uling procedures. The ongoing hand scorers 
left the special exercises blank. These exer- 
cises went through the normal machine proc- 
essing procedures and then were pulled from 
the packages and held until the time of the 
special scoring session. These sessions were 
held after nearly all the regular scoring and 
processing for the assessment had been com- 
pleted. After scoring, the exercise pages were 
rescanned and data tapes were created. The 
tapes were later combined to construct Na- 
tional Assessment respondent data files. 



Special Projects 

Each year, NAEP and MRC staffs also carried 
out various tryouts, feasibility studies and 
validity studies, which are an integral part of 
NAEP's development, quality-assurance and 
feedback program. These special studies 
varied depending on the requirements of the 
learning area being assessed. 
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CHAPTERS 
DATA PROCESSING 



The data gathered by Research Triangle Insti- 
tute (RTI) and Measurement Research Center 
(MRC) were combined, organized and ana- 
lyzed by National Assessment (NAEP). Speci- 
fications for analysed to be performed were 
supplied by NAEFs Research and Analysis 
Department staff. NAEP's Data Processing 
Department staff is responsible for providing 
fast, easy access to the data on a daily basis. 
They also engage in long-range planning and 
developmental work and provide general tech- 
nical support to all the project departments 
and their advisory committees. 



Computer Hardware and Languages 

National Assessment uses an IBM 360 Model 
20 computer located in Denver, an IBM 360 
Model 91 computer located at the Princeton 
University Computer Center and a high-speed 
communications link between the two com- 
puters. Card decks are read into an IBM 2560 
card reader in Denver and transmitted to the 
Model 91 at Princeton for execution* Print- 
outs are transmitted from Princeton to an 
IBM 1403 Printer in Denver. 

Because of the large amount of data collected 
by NAEP, the National Military Command 
Information Processing System (NIPS— a Data 
Base/ Data Management System) is used to 
process NAEP data, it provides a convenient 
method for solving the data-handling prob- 
lems associated with high-volume and large- 
file processing requirements by using execu- 
tive programs which simplify common opera- 
tions such as data definition, data storage, 
maintenance and retrieval; the amount of 
human effort required to code» test and debug 
is reduced. 



In addition to NIPS, various general-purpose 
programming languages, chiefly PL/I and 
FORTRAN are used. These languages are well 
suited for statistical computations as well as 
data manipulations on the various files. 



Preparation for an Assessment Year 

As the data were being collected in the field, 
files that would hold the data and the 
procedures that would be used to manipulate 
them were being designed at National Assess- 
ment. MRC and NAEP discussed specifica- 
tions for the format in which the data would 
be prepared by MRC and the edit checks that 
MRC would perform to insure correct, accu- 
rate data. To insure compatibility, formats 
were also determined for data from special 
scoring. 

Approximately three months before the first 
MRC-produced data tape was scheduled to be 
completed, file design and coding began for 
the respondent file. The magnetic tapes con- 
taining response data for each age level were 
received from MRC at six-to-eight-week inter- 
vals. The data was loaded onto the NAEP 
respondent files as it was received. Approxi- 
mately four weeks after the MRC tape for an 
age level was received, a tape containing a 
weighting factor for each respondent was 
received from RTI, and the weights were 
merged with the respondent file.^ 



^ Refer to Chapter 7, Computation of P* Values, for a 
discussion of weighting. 
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Organizing the Data 

A variety of files used to check the accuracy 
of the data and implement various steps in the 
analyses of the data were developedo 



Exercise Documentation File 

The exercise documentation file contained 
exercise-level information for each exercise, 
keyed by learning area, exercise number, 
overlap and part. The file included informa- 
tion about the parts of each exercise, such as 
a short description of the exercise part, the 
correct foil, the response range, specifications 
for combinations of parts and analysis specifi- 
cations. The file also contained specifications 
needed by the retrieval program used to 
create auxiliary files. 



Respondent File 

The respondent file was the main source of 
data used for all analyses. Four respondent 
files were created from the MRC and RTI 
tapps-^one for each age group. Each file was 
organized by package-exercise number and 
contained records keyed by age, package, PSU 
(primary sampling unit) and individual identi- 
fication number^ach respondent represented 
by one record. After the respondent data was 
loaded onto the files, the weighting factors 
were loaded along with data obtained from 
the principal's questionnaires or background 
questionnaires. 



Auxiliary Data Files 

Since each analysis used only a fraction of the 
data from the respondent file, smaller auxil- 
iary files were created in order to increase 
processing efficiency. Each auxiliary file 
represented one age level for one learning 
area. Part of the process of creating these files 
was to produce data fields which contained 
combinations of original items of data for 
each respondent. These additional, derived 
data fields were stored with the original data 
on the auxiliary files. 
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After the auxiliary files had been created, 
preliminary analysis programs computed per- 
centages of individuals choosing each response 
for each exercise. They were checked for the 
validity of response codings and the potential 
usefulness of the combinations of certain 
responses. If an invalid coding of a response 
was discovered, MRC was requested to locate 
the original package in order to correct the 
error. CoiTections of the data on the respond- 
ent file were made and revised specifications 
for analyses were prepared. New auxiliary 
files were created and final computer pro- 
grams were written to calculate percentages, 
standard errors and other statistics. 



Summary Files 

The statistics calculated from the auxiliary 
files were stored on exercise statistic summary 
files. One summary file was created for each 
learning area and was keyed by age, package, 
exercise, part and response. Each record 
contained the statistics which had been com- 
puted for a single response. Additional coded 
information indicating exercises with similar 
characteristics was also added to this file, to 
be used later in the computation of summary 
statistics across groups of exercises. 

Various summary analyses were performed on 
data from those exercises with similar charac- 
teristics, For example, the music performance 
exercises were summarized, and results for the 
social studies exercises dealing with political 
knowledge and attitudes were summarized. A 
variety of data display programs were used to 
present tables of medians, means, maxima and 
minima as well as graphic displays of the 
distributions of certain exercise statistics. 
Such summaries provided the basis for gener- 
alizations about national and subpopulation 
performances on various types of exercises. 



Appendix File 

For each learning area an appendix file was 
created to produce camera-ready printouts of 
response statistics for NAEP statistical re- 
ports. Descriptive text for each exercise 
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response was coded, keypunched and edited. 
This was a massive task. For example, music 
alone had over 17,000 different response 
possibilities. £xercise*level statistics were ex- 
tracted from the summary files and combined 
with the edited text to create the appendix 
files. 



Availability of Data 

In addition to statistical and prose reports 
produced from the various data analyses, 
National Assessment also produces computer 
data tapes for the learning areas assessed. 



These tapes are created from the respondent 
files and contain the original response data for 
the exercises which have been released into 
the public domain in their entirety. In order 
to protect anonymity of respondents and 
political units, the tapes were designed so that 
it is not possible to relate individual responses 
to particular respondents, schools, school 
districts or states. However, there is complete 
documentation of the classification systems 
and the contents of the tapes so that the data 
may be used and reanalyzed to suit the needs 
of researchers and other interested indi- 
viduals. 
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CHAPTER? 
DATA ANALYSIS 



National Assessment (NAEP) estimates the 
percentages of individuals at each of four age 
levels in the nation as a whole and in certain 
groups who possess various knowledges, 
understandings, skills and attitudes. 

To do so, the Assessment has developed two 
baseline measures: (1) the p-value, an estimate 
of the percentage of persons in each of the 
populations v/ho gave a ertain response to a 
specific exercise, and (2) the Ap-value (delta- 
p), an estimate of the difference between the 
performance of a subpopulation and the 
performance of the total age population. 
Measures of change will reflect comparisons 
of these statistics over time J 

The most frequently reported p-values are 
those that estimate the percentage oC a total 
age population or a certain subpopi^ nation 
that gave an acceptable response to an exer* 
cise. However, NAEP computed many p* 
values in addition to the percentage of accept- 
able responses. For multiple*choice exercises, 
percentages were computed for each unac- 
ceptable choice, the ^^I don't know" response 
and the total percentage of the unacceptable 
responses as well* Open-ended exercises gener- 
ally had several categories of acceptable and 
unacceptable:^ responses. Percentages were 
computed for each category as well as for the 
total number of acceptable and unacceptable 
responses. Not all exercises had clearly defin- 



detailed discussion on National Assessment's 
statistical computation procedures is available in 
Robert Larson, Wayne Martin, Todd Rogers, Don 
Searis. Susan Sherman and David Wright, A Look at 
the Analysis of National Assessment Data presented 
by J. Stanley Ahmann in Frontiers of Educational 
Measurement and Information Systems ~- 1973. ed. 
William E. Coffman (Boston: Houghton Mifflin Com- 
pany, 1973), pp. 89-111. 



able acceptable and unacceptable criteria; for 
example, some were survey questions, and 
estimates were made of the percentages of 
people who played certain kinds of musical 
instruments or read certain kinds of science 
books or magazines. 

Many exercises had several parts or required 
individuals to give more than one response, 
each of which could be scored acceptable/ 
unacceptable. Additional analyses summa- 
rized results over the parts of an exercise. On 
these multi-part exercises the percentage of 
individuals giving exactly zero, one, two, 
three, etc. acceptable responses were reported 
as well as the percentage of individuals giving 
at least one, at least two, at least three, etc. 
acceptable responses. 

For some multi-part exercises, analysis of 
combinations of responses from two or more 
parts provided additional interesting informa- 
tion. For example, the following combina- 
tions were derived: (1) percentages of indi- 
viduals who agreed or disagreed with a state- 
ment and successfully defended their posi- 
tions; (2) percentages of people who could 
not read music, but played a musical instru- 
ment acceptably; and (3) percentages of 
respondents who identified social problems, 
but could not propose solutions for those 
problems. 



Computation of P- Values 

The percent^es reported by NAEP are 
weighted estimates of population percentages. 
The procedures for developing the estimators 
and their standard errors are related to the 
sample design* If every individual had, by 
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design, an equal probability of being selected 
to respond to a NAEP exercise, a simple ratio 
of the number of successes to tlie number 
responding would be a good estimate of the 
actual percentage of acceptable responses. In 
that case, each individual in the sample would 
represent the same number of individuals in 
the population. However, in Years 03 and 04 
the Assessment determined that all 50 states 
would be represented in the sample. Also, to 
insure an adequate number of respondents in 
the sample at the lower end of the socio- 
economic scale, respondents were selected at 
a higher rate from this subpopulation. The 
**all states*' requirement and oversampling of 
low socioeconomic individuals resulted in 
individuals in the population having unequal 
probabilities of being selected. To adjust for 
this unequal rate for the sample, appropriate 
weighting of the data was used to produce 
unbiased estimates of the percentages. The 
national percentages— p -values for a total age 
group— were computed using the weighted 
responses of all respondents in the sample. 
For example, the estimated percentage of 
9-year-olds who answered acceptably is de- 
fined as the sum of the weighted number of 
acceptable responses divided by the sum of 
the weighted number of responses for all 
9-year*olds. Percentages for each reporting 
group were computed in the same manner 
using only that group's weighted responses. 



AP- Values 

A Ap-value is the difference between the 
estimated percentage of a group {for example, 
the Western region) that provided a particular 
response to an exercise and the corresponding 
national percentage: 

Western % — National 7r - Western Ap-value. 

If, on a given exercise, a group's percentage is 
lower than the national percentage, the differ- 
ence between the two percentages is ex- 
pressed as a negative number; if a group's 
percentage of success is higher than the 
national percentage, the group's difference in 
performance is a positive number. For exam- 



ple, if on a given exercise 74% of the 
13-year-olds in the West gave an acceptable 
response, but only 68% of the national sample 
did so, then the West Ap for 13-year-olds 
would be +6 percentage points. 



Reliable Differences 

The p-values are estimates of the population 
percentages, based on the percentages ob* 
tained from the sample. Somewhat different 
percentages would be obtained if a different 
sample had been chosen. Variation would 
occur among the percentages for all possible 
samples that could have been selected for 
Years 03 and 04. Since only one sample is 
drawn per year, an estimate of the variability 
among all possible values for samples of that 
size must be computed to determine how 
precise the obtained estimated percentages 
were. Since results are reported for each 
exercise , it is desirab le th at the vari an ce 
estimation procedures be inexpensive and yet 
yield unbiased, stable estimates. The jackknife 
proc*edure, a computationally simple and rela- 
tively inexpensive procedure, was used in 
Years 03 and 04,^ The measure of variability 
computed is the standard error. 

When comparing a group with the nation, it 
becomes important to decide how large the 
sample estimate of the difference must be in 
order to be certain that the difference is not 
zero, The project has adopted a statistical 
convention that if an obtained difference is at 
least twice as large as the estimate of its 
standard error, then the Ap is said to be 
reliably different from zero. Statistically, this 
sort of statement has a 95% probability of 
being correct. Such differences are called 
statistically significant and are indicated on 
exhibits in reports by asterisks (*). 

Of course, the practical importance of differ- 
ences to the educational community is not 

^P. Mosteller and J.W. Tukey, ••Data Analysis, 
Including Statistics/' in Handbook of Social Piiychnh 
ofty (2nd ed.)» eds. G. Lindzoy and E. Aron8on 
(Reading, Mass.: Addison-We&ley, 1968), pp. 
80-203. ^ 
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necessarily determined by statistical relia- 
bility. The practical importance of larger or 
smaller differences must be judged by the 
reader of NAKP reports. 



ance can be displayed by presenting its 
median difference and the range of the middle 
50% of its exercise differences (see Exhibit 
7-1). 



Summar/ Results 

Each learning area is capable of generating a 
very large number of percentage values. To 
help readers of NAEP reports, results must be 
summarized in a general way as well as 
presented in detail individually. National 
Assessment summarizes data for clusters of 
exercises that would be of public interest. For 
example, in addition to exercises classified as 
political knowledge and attitudes and those 
classified as music performance in Year 03, 
summary analyses will be performed on clus- 
ters of math and science exercise^. 

For a given set of exercises, a group's achieve- 
ment is summarized conveniently by using its 
differences from national percentages of suc- 
cess. For example: If on a set of 5 exercises 
the percentages of success for all 9-year-olds 
were 90%, 40%, 82%, 75% and 60%, and the 
percentages of success for a particular 
group— Northeast 9-year-olds, for instance— on 
these same exercises were, respectively, 95%, 
44%, 85%, 77% and 61%, then the group's 
differences would be +5%, +4%, +3%, +2% 
and +1% . The group's median difference from 
a national performance leveHthat figure 
above and below which 50% of its exercise 
differences lie— would be +3%. If more than 
12 exercises are summarized the median 
difference provides a stable indicator of a 
group's typical performance over the set of 
exercises. If one desires a single figure to 
describe a group's performance relative to a 
national level of performance on a set of 
exercises, this is clearly the most useful figure 
to consult. However, a more complete picture 
of a group's typical performance emerges 
from examination of the entire range of 
differences or, more conveniently, the range 
of the middle 50% of the exercise differences. 
For example, given a full range of exercise 
differences (from +12% to —14% in a hypo- 
thetical example) the group's typical perform- 



EXHIBIT 71. How Differences from the National 
Percentage of Success Are Reported: Sample Graph 
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If a large proportion of the group*s differ- 
ences occur above (or below) the national 
level, there is an even stronger indication of 
how that group's performance differs from 
the national level. To examine the group's 
extreme differences, the top 25% and bottom 
25% of the distribution of Ap-values can be 
useful. If a group's performance on any 
individual exercise deviates considerably from 
the patterns established in the overall sum- 
mary data» the deviation could be of special 
interest. 

Results for certain clusters of exercises in 
each learning area have been compared in 
various ways. For example, the median differ- 
ence for exercises clustered together in one 
reporting topic may have been 8 percentage 
points below the national level for 13-year- 
olds in the Southeast. For a different cluster 
of exercises that group's median difference 
may have been 5 percentage points above the 
nation. Therefore, relative to the nation the 
group tended to perform better on one cluster 
than on the other. 



Or* 
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Estimation of Change 

The NAEP procedure for determining change 
is to reassess certain exercises (approximately 
half of those assessed in each learning area) 
four or five years after the previous assess* 
ment and compute differences in performance 
on those exercises. There are at least two 
straightforward measures of change that will 
be computed for each exercise: 

1. the differences in percentage of correct 
responses on a given exercise from one 
assessment to the next and 

2. the differences in group Ap-values on a 
given exercise from one assessment to 
the next. 

Data from the first reassessment (science) are 
currently being analyzed and will be available 
in the science change report to be published 
in 1975. 



Limitations of the Data 

Within the limitations due to measurement 
and sampling enor, the data computed and 



reported for Years 03 and 04 accurately 
describe the educational achievements of the 
groups designated in the sample. 

When the data show that a group^s overall 
level of achievement is either above or below 
the national level, one must exercise great 
caution in speculating about the causes. Con- 
sider, for example, a hypothetical group 
whose achievement is well above the national 
performance level. Most individuals classified 
as members of the group may attend schools 
that have excellent physical facilities and 
high-quality faculties, belong to families that 
have attained a high socioeconomic level, have 
well*educated parents and come from homes 
with many reading materials. Any of these 
factors could contribute to the group's high 
level of achievement, while membership in the 
group itself may contribute very little or 
nothing. When one looks at the data for a 
given group, therefore, it cannot be said that 
any difference in achievement between that 
group and the nation as a whole is attrib* 
utable solely to membership in that group. 
Interpretation of results must take into 
account the other factors which might be 
influencing the achievement of the group. 
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CHAPTER 8 

REPORTING AND DISSEMINATING THE FINDINGS 



National Assessment (NAEP) generates a great 
deal of educational information which then 
must be widely communicated. Given the 
broad scope of the project and the variety of 
audiences NAEP serves, this is at best a 
difficult task. 

Various NAEP materials represent different 
organizations of content and levels of detail. 
Some publications are highly technical and 
oriented to the research community; some are 
less detailed and focus on selective issues or 
concerns; others, like this yearbook, provide 
the general background necessary to under* 
stand and proptrly use NAEP results. 

At the most technical level are the statistical 
reports which provide all the information 
NAEP has gathered in a particular learning 
area. These reports present each released 
exercise exactly as it was presented to the 
respondents and documented to the extent 
that any secondary user of National Assess- 
ment exercises (e.g., a state assessment) can 
administer it exactly as it was originally 
administered. For released exercises, percent- 
ages appear for the acceptable responses (or 
categories), unacceptable responses (or cate- 
gories), the **I don't know" response and no 
response. Unreleased exercises are neither 
shown nor completely documented; the data 
are usually restricted to the correct response, 
don't know" response, no response and all 
incorrect responses grouped together. All per- 
centages are accompanied by their standard 
errors. This statistical report, which also 
contains summary data, is intended primarily 
for people involved in state and local assess- 



ments, educational researchers, educational 
decisionmakers and any other groups or 
persons who may need or desire complete and 
documented data. Other publications of this 
nature include special monographs on sam- 
pling and background variables. 

Selective reports, on the other hand, deal with 
timely, important or interesting aspects of a 
learning area in a nontechnical way. An 
overview report in each learning area provides 
a broad view of the findings in popular 
language with some specific, interesting 
details highlighted. These reports are intended 
primarily for professional associations, 
subject-matter groups, civic groups, public 
officials at local, state and national levels and 
those educational decisionmakers and any 
other persons or groups who may neither 
need nor desire complete and documented 
data. 

The Assessment does not attempt to interpret 
its findings or discuss their implications for 
the educational community in any of its 
reports. Under the auspices of National As- 
s essm ent, however, various professional 
groups or panels of professional individuals 
are writing interpretive reports. For instance, 
the National Science Teachers Association has 
published an interpretive study of the results 
of the first science assessment, and the Na- 
tional Council of Teachers of Mathematics 
plans to do the same in 1974^75. The 
interpretations, implications and evaluations 
in these reports are solely those of the 
professional groups or panels providing them 
and do not reflect NAEP's point of view. 



Finally, the project's findings appear in its 
Newsletter, in press releases and newspaper 
columns, in the publications of professional 
groups such as National Council of Teachers 
of English, Music Educators National Confer- 
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ence or National Council of Teachers of 
Mathematics, in general education publica- 
tions such as American Education or Compact 
and even in magazines such as Reader's 
Digest 
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CHAPTERS 

ADAPTATION AND UTILIZATION OF THE NATIONAL ASSESSMENT MODEL 



Although data is gathered on a national level, 
National Assessment (NAEP) recognizes the 
potential importance of its model, technology 
and materials as they relate to the solution of 
problems at state and local levels. For this 
reason, NAEP has created the Model 
Utilization/ Adaptation (MU/A) program to 
encourage and facilitate the adaptation of the 
Assessment model, technology and materials 
(or selected aspects of these) by state and 
local educational systems. To the extent that 
this is done, questions regarding the educa- 
tional outcomes being answered at the na- 
tional level can be answered in a more refined 
sense at state and local levels. The MU/A 
program provides services to state and local 
educational systems in three ways: 

1. consultation between MU/A staff and 
state or local officials to determine the 
feasibility of adapting the NAEP model 
to meet local needs, 

2. technical assistance to state and local 
officials in the use of the Assessment 
model or certain aspects of it and 

3. workshops and seminars that provide an 
exchange of ideas between NAEP and 
state and local assessments. 

Because the project does not presently have 
the resources to deal with the many diverse 
requests from local educational systems, these 
MU/A services can be made available to 
officials at the local level only on a limited 
basis unless special circumstances dictate 
otherwise. Generally, the MU/A program does 
no more than provide available materials to 
local educational systems and communicate 
with them by telephone or mail about general 
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problems, referring them to their state depart- 
ments of education for information about 
content-referenced assessment* In certain 
instances, further consultation, technical as- 
sistance and participation in workshops and 
seminars are made available to local educa- 
tional systems: 

1« if the local assessment is an integral part 
of a state assessment plan involving the 
adaptation of NAEP procedures and 

2. if NAEP foresees the possibility for the 
demonstration of a new form of adapta- 
bility that could provide valuable in- 
sights into significant uses of NAEP 
procedures at the local level* 

The MU/A program provides consultation to 
officials not directly concerned with assess- 
ment through a state educational system or a 
local educational system only when a clear 
and distinct possibility of improving educa- 
tional decision making can be established in 
preliminary discussions. 



Consultation 

Upon request from state officials, the MU/A 
program provides consultation to state educa- 
tional agencies, state boards of education, 
state legislative groups and statewide citizen 
groups. 

In providing consultation through its MU/A 
program. National Assessment does not actu- 
ally implement state (or local) assessments, 
provide prefabricated assessment plans nor 
endorse any state (or local) assessment. Con- 
sultation is provided as input to assist an 
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educational agency in determining the feasi- 
bility of adapting the NAEP model, tech- 
nology or materials to the agency's specific 
assessment needs. 

Some of the questions that might be discussed 
by the MU/A staff and state officials are: 

1. What reporting categories best fit the 
state's needs? 

2. If sampling is to be used, what sample 
design is most efficient both in terms of 
cost and obtaining the requisite data for 
the reporting categories? 

3. What analyses best fit the sample design 
and reporting needs? 

4. Should the state use all of NAEP's 
exercises, just the easily administered 
and easily scored multiple-choice exer- 
cises, just those pertaining to certain 
objectives or just a portion of those 
pertaining to each objective? 

Technical Assistance 

If the MU/A consultation with state officials 
has been fruitful and the state has decided to 
adapt the NAEP model, technologies or mate- 
rials for its assessment, the MU/A program 
provides continued technical assistance. Once 
a state has decided that it is feasible to adapt 
some aspect(s) of the project, the MU/A 
program, through technical assistance, helps 
the state to inaugurate and carry out the 
adaptations to assure the best possible chance 
of success. 

Technical assistance is generally provided 
directly by the MU/A staff. However, depend- 
ing upon the nature and degree of expertise 
required, other NAEP staff or outside consult- 
ants may be required to provide the assist: 
ance. For example, a state particularly con- 
cerned about the implications of sampling 
may require the technical assistance from the 
NAEP sampling coordinator. 



Workshops and Seminars 

Because the MU/A program is relatively new 
and because the entire area of content- 
referenced assessment at state and local levels 
is alL»o new and constantly evolving, the 
program "elies heavily upon the dissemination 
and exchange of ideas through personal con- 
tact. To facilitate this dissemination and 
exchange of ideas, NAEP conducts an annual 
assessment workshop. Most of the partici- 
pants represent state educational agencies. 

Currently, two types of workshops are being 
conducted. One type is for participants who 
are relatively new to the field of educational 
assessment. Its primary purpose is to intro- 
duce approaches developed by NAEP. Exam- 
ples of some of the topics covered in these 
workshops include discussions on how the 
NAEP model can be adapted for the types of 
data that might be provided for educational 
decision making. This allows the participants 
to examine the ways in which such a model 
could be adapted to the needs of their various 
states. This workshop also provides the parti- 
cipants with an opportunity to become aware 
of other services (consultation and technical 
assistance) the project makes available to 
states. 

The second type is a conference workshop for 
those already exposed to educational assess- 
ment. This conference allows the partici- 
pants to discuss their experiences with state 
assessment during the year, to gain insight 
from the experiences of other states and to 
leam of any changes in the NAEP model. 

Specialized seminars concerning sampling and 
analysis, reporting disseminating and utilizing 
and applying assessment data are anticipated. 
Previous experiences have shown these topics 
to be of great interest to state department 
personnel planning and conducting assess- 
ments. These seminars will be limited to 
participants involved in the sampling/analysis 
or reporting/dissemination/application phases 
of assessment in order that more detailed 
information and personal interaction can be 
provided. 
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As state and local educational agencies de- 
velop and implement assessment plans adapt- 
ing the Assessment model, technologies and 
procedures, the MU/A program develops 
niaterials to fulfill two purposes: 

1. documentation of the plans and metho- 
dologies for implementation, including 
both success and problem areas, for 
those who plan later assessments and 



2. documentation of the utility of ana- 
lyzed and interpreted data from state 
and local assessments to show other 
educational agencies the usefulness of 
assessments based upon adaptations of 
various components of the NAEP 
model. 
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APPENDIX A 

DEFINITIONS OF NATIONAL ASSESSMENT REPORTING POPULATIONS 



National Assessment (NAEP) reporting popu- 
lations include 9, 13 and 17-year-olds enrolled 
in public or private schools as well as 17-year- 
oids not enrolled in school and young adults 
aged 26 to 35. Within these age groups, results 
are reported by geographic region, sex, color, 
parental education and size and type of 
community. 

With a few exceptions, listed below, the 
reporting populations include all age-eligible 
persons in the 50 states and the District of 
Columbia. Nine and 13*y ear-old populations 
are restricted to those students enrolled in 
public or private schools during the survey 
period. The 17-year-old population includes 
students enrolled in public or private schools 
during the survey period, those age-eligible 



persons not enrolled but identified by schools 
as having dropped out or graduated early or 
those age*eligible persons not enrolled but 
living in identifiable housing units in the 
United States. Survey periods and age ranges 
for the four age groups are shown in Exhibit 
A-1. Age-eligible persons are excluded if they 
are: 

1. non-English speaking, 

2. institutionalized and 

3. physically, emotionally, or mentally 
handicapped in such a way that they 
cannot respond to the exercises as 
administered. 



EXHIBIT A-I. National Assessment 
Populations, Survey Periods and Age Ranges 



Age Level 



9 
13 

1 7 in-school 

1 7 not enrolled in school 
during January 1972 

1 7 not enrolled in school 
during March 1971 

Young adults (Year 03) 

Young adults (Year 04) 



Year 03 
Survey Period* 

January - February 1972 
October - December 1971 
March -April 1972 

January - July 1972 



Year 03 
Birth*Date Range* 

1/1/62 - 12/31/62 
1/1/58 12/31/58 
10/1/54 - 9/30/55 

10/1/54 9/30/55 



January - July 1972 10/1/53 - 9/30/54 

January ~ July 1972 4/1/36 - 3/31/46 

October 1972 May 1973 1/1/37 ^ 12/31/46 



*Ff>r Year 04 survey periods and birth-date ranges of 9, IS and 1 7'y earmolds, add 
one year to the Year 03 dates. < 
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National Assessment Groups 



Geographic Regions 

Results are reported by the four geographic 
regions defined by the Office of Business 
Economics, Department of Commerce. The 
states in each region are shown in Exhibit 
A-2. 



EXHIBIT A-2. Definitions of National 
Assessment Regional Subpopulations* 



Northeast 


Southeast 


Delaware 


Arkansas 


C\)nnt?ctiL!Lit 


Plnridn 




Viminia 


\ew Haninshire 


Wpct Viroinia 


Rhode Kl^nd 


rZdla l/ulll u 


Vermont 


Georgia 


District of Columbia 


Kentucky 


Maryland 


Louisiana 


Massachusetts 


Mississippi 


New Jersey 


North Carolina 


Pennsylvania 


South Carolina 


New York 


Tennessee 


Central 


West 


Iowa 


Alaska 


Kansas 


Hawaii 


Nebraska 


Idaho 


North Dakota 


Montana 


South Dakota 


Nevada 


Minnesota 


Wyoming 


Missouri 


Arizona 


Illinois 


Oregon 


Indiana 


Utah 


Michigan 


Colorado 


Wisconsin 


New Mexico 


Ohio 


Oklahoma 




California 




Texas 




Washington 



*These regional suhpopulation definitions are the 
same as those used by the Office of Business 
Economics, Department of Commerce. 



Sex 

Results are reported separately for males and 
females at all age levels. 



Color 

Results are reported for Whites and Blacks. 
While respondents were classified into several 
other ethnic groups, including Puerto Ricans, 
Mexican-American and other or unclassified, 
the actual sample sizes realized were insuf- 
ficient for reporting purposes. 



Parental Education 

Parental education refers to the highest level 
of education level reported by the respondent 
for either parent. 

No high school. Neither parent has any formal 
education beyond the eighth grade. 

Some high school. At least one parent has 
some formal education beyond the eighth 
grade, but neither parent has graduated from 
high school. 

Graduated from high school. At least one 
parent has graduated from high school, but 
neither parent has any formal education 
beyond high school. 

Post high school. At least one parent has some 
formal education beyond high school in- 
cluding any business, professional or trade 
school training as well as college or university 
training. 



Size and Type of Community (STOC) 

The seven size and type of community 
(STOC) reporting categories are comprised of 
three **extreme" types of community (TOC) 
and four "residual" sizes of community 
(SOC). Each TOC category includes approxi- 
mately 10% of the respondents at each age 
level; the remaining respondents are classified 
according to one of the SOC classifications. 

Briefly, the three TOC categories are: (1) city 
areas where a high proportion of the adult 
population is either not regularly employed or 
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on welfare and a low proportion is employed 
in professional or managerial positions; (2) 
rural areas where a high proportion of adults 
are farm workers and a low proportion are 
professional, managerial or factory workers; 
and (3) near-city and city areas where a high 
proportion of adults are employed in profes- 
sional or managerial positions and a low 
proportion are factory or farm workers, not 
regularly employed or on welfare. Respond- 
ents are placed in one of these categories if 
the occupational profile and location of the 
school or, in the case of the out-of-school 
sample, segment satisfy the extreme TOC 
definitions. 

The remaining respondents at each age level 
are classified according to the size of commu- 
nity in which the school or segment is 
located. The occupational profile is based on 
the employment categories summarized in 
Exhibit A-3. 

For the in-school sample at each age and the 
supplementary sample at age 17, the school 
principal of each selected school provided 
estimates of the percentage of students whose 
parents fit into each occupational category. 



Other occupational data for the out-of-school 
sample was obtained bom census data in Year 
03 and from the respondents themselves in 
Year 04. The definitions used to classify 
respondents by STOC are presented in 
Exhibit The occupational index is com- 
puted using the occupational categories sum* 
marized in Exhibit A*3. The distribution of 
respondents by STOC category for Year 03 is 
presented in Exhibit A*5. 



EXHIBIT A-3. Occupational Categories 



Categories Code 

Professional or managerial A 
personnel 

Sales, clerical, technical 3 
or skilled workers 

Factory or other blue collar C 
workers 

Farm workers D 

Not regularly employed E 

On welfare F 
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EXHIBIT A-4. National Assessment Si !e and Type of 
Community (STOC) Reporting Categnies 



Reporting 
Category 

Low metro 



Extreme rural 



Small place 



Medium city 



Main big city 



Occupational 
Index* 

E+F-A 



D-(C+2A) 



Description 



Sample schools or segments in a city or metropolitan 
area of a city with a population greater than 
1 50,000 and in the 90-99th percentile of the low 
metro index 

Sample schools or segments in communities with a 
population less than 8,000 and in the 90-99th 
percentile of the extreme rural index 

Sample schools or segments in a community with 
a population less than 25,000 and not classified 
as extreme rural 

Sample schools or segments in a city with a 
population between 25,000 and 200,000 and not 
classified as low metro or high metro 

Sample schools or segments within the city 
limits of a city with a population greater than 
200,000 and not classified as high metro ot 
low metro 



Urban fringe 



High metro 



A-(C+D+E^^F) 



Sample schools or segments in the metropolitan 
area of a big city but outside the city limits 
and not classified as low metro, extreme rural 
or higii metio 

Sample schools or segments in a city or 
metropolitan area of a city with a population 
greater than 150,000 and in the 90-99lh 
percentile on the high metro index 



*See Exhibit A-J. 



EXHIBIT A<5. Distribution of Respondents by Size 
and Type of Community (STOC) and Age Level 

STOC Category Age Level 







9 


13 


17 


17 


17 


Adult 










(IS)* 


(OS)t 


(AU) 




1. 


Low metro 


10.2% 


9.9% 


11.3% 


14.0% 


11.7% 


9.9% 


2. 


Extreme rural 


10.0 


9.6 


11.8 


9.8 


1 1.6 


10.2 


3. 


Small place 


32.4 


33.7 


33.0 


33.4 


33.1 


31.8 


4. 


Medium city 


16.8 


16.6 


15.9 


19.9 


16.3 


13.7 


5. 


Main big city 


lO.S 


11.2 


9.3 


6.4 


9.0 


8.0 


6. 


Ui'ban fringe 


10.0 


8.4 


5.1 


7.7 


5.4 


16.4 


7. 


High metro 


10.1 


10.7 


13.6 


8.9 


13.0 


10.0 



*In'School 
fOuhof-school 
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APPENDIX B 

ESTIMATED POPULATION DISTRIBUTIONS 
OF NATIONAL ASSESSMENT REPORTING GROUPS 



Based on Year 03 data, Exhibits B-1 through 
B-4 contain the percentage of the population 
at each age level estimated to be in each 
two-w{^ combination of the reporting groups 
defined in Appendix A. These data provide a 
more complete description of the composi- 
tion of the population at each age level in 
terms of National Assessment's (NAEP) re- 
porting variables. Each entry is an estimate of 
the percent of a reporting group identified by 
a row heading found in a second reporting 
group identified by a column heading. For 
example, in each exhibit, the rows corre- 
sponding to color^Black, White, Other-show 
the estimated percentages of each color group 
in the nation, each region, sex, parental 



education level and age and type of com- 
munity classification. As shown in Exhibit 
B-1, the estimated percentage of 9-year-old 
Blacks (row 7) in the nation (column 1) is 
13.6%, while of the total population of 
9-year-old8 in the Southeast (column 2), 
26.6% are Black (row 7). The estimated 
percentage of 9-year-old Blacks living in the 
Southeast, 45.5%, is shown in row 1, column 
8. The exhibits have been arranged so that for 
each reporting group identified by a column 
heading the percentages across the levels of a 
reporting variable identified by rows add to 
100%. For example, the sum of the percent- 
age of Blacks, Whites ana Other equal 100% 
in every column of the exhibits. 
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EXHIBIT B-la 

Estimated Population Distributions of National Assessment 
Reporting Groups (9-Year-01d3) 



Variables 
and Groups 



Natl. 



SE 



Region C%) 



U 



NE 



Sex (S) 



Color (g) 



W 



0th. 



Region 
Southeast 
West 
Central 
Northeast 



23.2 
24.0 
27.8 
25.1 



100.0 
0.0 
0.0 
0.0 



0.0 
100.0 
0.0 
0.0 



0.0 
0.0 
100.0 
0.0 



0.0 
0.0 
0.0 
100.0 



22.7 
23.5 
28.2 
25.7 



23.7 
24.5 
27.4 
24.5 



45.5 
18.9 
18.8 
16.8 



21.1 
20.5 
31.2 
27.4 



1.2 
81.2 

3.9 
13.7 



Sex 

Male 
Female 



49.1 
51.1 



47.9 
52.1 



47.9 
52.1 



49.7 
50.3 



50.2 
49.8 



100. 0 
0.0 



0.0 
100.0 



47.0 
53.0 



49.5 
50.6 



47.5 
52.5 



Color 
Black 
White 
Other 



13.6 
80.4 
6.2 



26.6 
73.1 
0.3 



10.7 
68.6 
20.8 



9.2 
90.0 

0.9 



9.1 
87.6 
3.4 



13.0 
81.1 
6.0 



14.1 
79.7 
6.3 



100.0 
0.0 
0.0 



0.0 
100.0 
0.0 



0.0 
0.0 
100.0 



Parental education 
No high school 
Some high school 
Graduated high school 
Post high school 
Unknown 



5.5 


9.5 


6.0 


4.2 


2.8 


5.4 


5.5 


7.2 


A. 6 


13.8 


4.9 


7.5 


4.1 


5.0 


3.0 


5.0 


4.7 


6.7 


4.5 


5.6 


23.4 


25.3 


23.0 


25.4 


19.7 


24.6 


22.3 


23.7 


23.7 


18.8 


35.2 


30.3 


38.5 


33.9 


37.7 


36.7 


33.6 


27.1 


37.6 


20.7 


31.3 


27.5 


28.4 


31.6 


36.8 


28.3 


33.9 


35.3 


29.7 


41.0 



Size and type of 
community (STOC) 
Lov aetro 
Extreme rural 
Small place 
Medium city 
Main hig city 
Urban fringe 
High metro 



8.9 


9.4 


11.9 


8.5 


5.7 


8.5 


9.2 


44.5 


1.8 


22.3 


8.6 


10.7 


10.5 


11.2 


1.7 


8.7 


8.4 


5.6 


8.8 


12.3 


29.7 


36.4 


23.1 


27.0 


32.6 


29.3 


30.1 


15.3 


32.6 


23.3 


16.8 


24.2 


15.6 


19.8 


7.8 


17.0 


16.6 


14.5 


17.6 


11.6 


10.5 


5.9 


9.9 


11.0 


14.5 


10.2 


10.6 


12.6 


9.7 


15.9 


11.5 


6.7 


13.9 


10.7 


14.4 


11.8 


11.2 


4.9 


12.7 


9.3 


14.3 


6.8 


15.1 


11.7 


23.3 


14.7 


13.9 


2.7 


17.0 


5.3 
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EXHIBIT B--lb 



Variables 
and Groups 



Region 
Southeast 
West 
Central 
Not^theasi; 



Estimated Population Distributions of National Assessment 
Reporting Groups (9-Year-01d8) 



Parental Education iZ) 



Size and Type of Community {%) 



NUS 


SHS 


GHS 


PHS 


Unk. 


LM 


ER 


SP 


MC 


MBC 


UF 


HH 


40.0 


35.9 


25.1 


20.0 


20.4 


24.6 


29.0 


28.4 


33.4 


13.1 


13.5 


11.1 


26.2 


20.1 


23.6 


26.3 


21.9 


32.4 


29.4 


18.7 


22.3 


22.8 


29.1 


25.4 


21.1 


28.4 


30.2 


26.8 


28.2 


26.7 


36.5 


25.3 


32.8 


29.4 


26.0 


22.8 


12.8 


15.7 


21.1 


26.9 


29.6 


16.3 


5.1 


27.6 


11.6 


34.8 


31.5 


40.8 



Sex 
Male 
F«imale 



48.6 
51.4 



50.5 
49.5 



51.5 
48.5 



51.2 
48.8 



44.5 
55.5 



47.0 
53.0 



49.7 
50.3 



48.3 
51.7 



49.6 
50.5 



48.0 
52.0 



50.2 
49.8 



50.3 
49.7 



Color 
Black 
White 
Other 



17.8 18.8 
66.8 74.1 
15.5 7.1 



13.7 
81.4 
4.9 



10.5 15.3 
85.9 76.7 
3.6 8.1 



68.2 3.8 

16.3 82.3 
IS. 5 8.9 



7.0 
88.2 
4.8 



11.6 
84.2 
4.2 



16.3 
74.4 
9.4 



5.8 2.5 
89.2 95.2 
S.O 2.3 



Parental education 



No high school 


100.0 


0.0 


0.0 


0.0 


0.0 


7.2 


10.4 


6.8 


5.6 


2.9 


4.2 


1.5 


Some high school 


0.0 100.0 


0.0 


0.0 


0.0 


5.3 


6.7 


6.0 


5.2 


3.3 


4.5 


1.9 


Graduated high school 


0.0 


0.0 


100.0 


0.0 


0.0 


21.3 


28.2 


26.9 


23.7 


19.9 


23.7 


16.4 


Post high school 


0.0 


0.0 


0.0 


100.0 


0.0 


27.2 


24.8 


32.5 


34.6 


31.4 


34.4 


55.5 


Unknown 


0.0 


0.0 


0.0 


0.0 


100.0 


38.9 


29.9 


27.8 


30.8 


42.5 


33.2 


24.8 



Size and type of 
community (STOC) 
Low metro 
Extreme rural 
Small place 
Medium city 
Main big city 
Urban fringe 
High metro 



11.6 


9.7 


8.0 


6.9 


11.0 


100.0 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


16.2 


11.8 


10.3 


6.0 


8.2 


0.0 


100.0 


0.0 


0.0 


0.0 


0.0 


0.0 


36.9 


36.9 


34.1 


27.4 


26.4 


0.0 


0.0 


100.0 


0.0 


0.0 


0.0 


0.0 


17.3 


18.2 


17.0 


16.6 


16.6 


0.0 


0.0 


0.0 100.0 


0.0 


0.0 


0.0 


5.4 


7.2 


8.9 


9.3 


14.2 


0.0 


0.0 


0.0 


0.0 100.0 


0.0 


0.0 


8.8 


10.7 


11.6 


11.2 


12.2 


0.0 


0.0 


0.0 


0.0 


0.0 100.0 


0.0 


3.8 


5.6 


10.0 


22.6 


11.3 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 100.0 
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EXHIBIT B-2a 



BEST COPY milABiE 



Escloiated Population Distributions of National Assessment 
Reporting Groups (IS-Year-Olds) 



Variables 
and Groups 



Natl. 

a) 



SE 



Region (%) 



W 



NE 



Sex (%) 



M 



Color (%) 



W 



0th. 



Region 
Southeast 
West 
Central 
Northeast 



22.6 
24.1 
29.4 
24.1 



100.0 
0.0 
0.0 
0.0 



0.0 
100.0 
0.0 
0.0 



0.0 0.0 

0.0 0.0 

100.0 0.0 

0.0 100.0 



22.8 
24.1 
29.3 
24.0 



22.4 
24.1 
29.5 
24.2 



46.1 
12.7 
23.8 
17.5 



21.1 
21.7 
31.8 
25.6 



1.9 
79.3 

5.0 
13.9 



Sex 
Male 
Female 



50.0 
50.1 



50.5 
49.5 



49.9 
50.1 



49.9 
50.2 



49.7 
50.3 



100.0 0.0 
0.0 100.0 



46.7 
53.3 



50.5 
49.7 



49.5 
SO. 5 



Color 
Black 
White 
Other 



10.3 
83.9 
5.8 



21.1 
78.5 
0.5 



5.4 
75.6 
19.0 



8.4 
90.7 
1.0 



7.5 
89.2 
3.3 



9.7 
84.7 
5.7 



11.0 
83.2 
5.8 



100.0 
0.0 
0.0 



0.0 
100.0 
0.0 



0.0 
0.0 
100.0 



Parental education 



No high school 


6.5 


10.2 


7.4 


5.4 


3.4 


6.4 


6.5 


9.8 


5.1 


19.9 


Some high school 


9.2 


14.0 


8.3 


8.7 


6.1 


8.2 


10.2 


16.2 


8.2 


11.2 


Graduated high school 


31.5 


30.7 


25.5 


37.6 


30.4 


31.4 


31.5 


29.1 


32.5 


20.9 


Post high school 


41.1 


33.8 


46.5 


38.8 


45.2 


41.6 


40.6 


21.9 


44.8 


21.1 


Unknown 


11.9 


11.3 


12.3 


9.6 


15.0 


12.4 


11.2 


23.0 


9.4 


26.9 



Size and type of 
community (STOC) 
Low metro 
Extreme rural 
Small place 
Medium city 
Main big city 
Urban fringe 
High metro 



7.3 


6.0 


8.5 


8.1 


6.1 


7.2 


7.3 


26.9 


3.6 


25.1 


8.3 


10.9 


6.7 


13.1 


1.6 


8.2 


8.5 


8.3 


8.4 


6.8 


32.4 


40.4 


26.7 


31.8 


30.8 


33.0 


31.7 


23.8 


34.1 


22.7 


16.2 


22.2 


15. S 


16.0 


11.2 


16.0 


16.3 


14.2 


16.6 


13.1 


11.6 


10.5 


7.9 


16.0 


10.7 


11.4 


11.7 


17.6 


10.7 


13.2 


10.9 


2.9 


17.3 


5.4 


18.4 


10.9 


10.8 


4.4 


11.5 


12.8 


13.8 


7.2 


17.4 


9.6 


21.? 


13.6 


13.9 


4.8 


15.4 


6.4 



4$ 



BEST COPY AVAIUBLl 



EXHIBIT B-2b 



Variables 
and Groups 



Estimated Population Diatributioiis of National Assessment 
Reporting Groups (IS-Year-Olds) 



Parental Education (%) 
NHS SHS GHS PHS Unk. 



Size and Type of Community 



LM ER 



SP 



MC 



MBC UF 



HM 



Region 
Southeast 
West 
Central 
Northeast 



35. 5 34.5 

27.6 21.9 
24.3 27.8 

12.7 15.9 



22.0 
19.6 
35.2 
23.3 



18.6 21.5 
27.2 24.7 

27.7 23.6 
26.5 30.3 



18.7 29.5 

28.3 1? ■• 

32.7 46.^ 

20.3 4.6 



28.2 
19.9 
28.9 
23.0 



31.0 
23.2 
29.1 
16.8 



20.5 
16.4 
40.8 
22.4 



6.1 11.8 

38.3 30.5 

14.7 20.5 

40.9 37.2 



Sex 
Male 
Female 



49.7 44.5 49.8 50.7 52.5 
50.3 55.5 50.2 49.4 47.6 



49.6 49.0 51.0 49.4 49.2 50.2 49.4 
50.4 51.0 49.0 50.6 50.8 49.8 50.6 



Color 
Black 
White 
Other 



15.7 18.2 

66.6 74.7 

17.7 7.1 



9.6 
86.6 
3.8 



5.5 19.9 
91.5 67.1 
3.0 13.0 



38.4 10.3 
41.7 85.0 
20.0 4.7 



7.6 
88.4 
4.1 



9.1 
86.2 
4.7 



15.8 
77.7 
6.6 



4.2 3.6 
89.0 93.7 
6.8 2.7 



Parental education 



No high school 


100.0 


0.0 


0.0 


0.0 


0.0 


10.5 


14.5 


7.5 


5.5 


5.8 


3.5 


1.0 


Some high school 


0.0 


100.0 


0.0 


0.0 


0.0 


11.4 


12.8 


11.1 


9.1 


9.8 


6.9 


2.5 


Graduated high school 


0.0 


0.0 


100.0 


0.0 


0.0 


29.3 


35.6 


34.8 


30.9 


32.5 


33.7 


20.2 


Post high school 


0.0 


0.0 


0.0 


.1.00.0 


0.0 


23.5 


26.1 


36.8 


43.4 


33.2 


45.2 


69.8 


Unknown 


0.0 


0.0 


0.0 


0.0 


100.0 


25.4 


11.1 


9.8 


11.2 


18.7 


10.7 


6.5 



Size and type of 
cosuminlty (STOC) 
Low metro 
Extreme rural 
Small place 
Medium city 
Main big city 
Urban fringe 
High metro 



11.7 


9.0 


6.8 


4.1 


15.4 


100.0 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


18.6 


11.6 


9.4 


5.3 


7.7 


0.0 


100.0 


0.0 


0.0 


0.0 


0.0 


0.0 


37.6 


39.3 


35.7 


29.0 


26.5 


0.0 


0.0 


100.0 


0.0 


0.0 


0.0 


0.0 


13.8 


15.9 


15.8 


17.1 


15.1 


0.0 


0.0 


0.0 


ICO.O 


0.0 


0.0 


0.0 


10.3 


12.3 


11.9 


9.3 


18.1 


0.0 


0.0 


0.0 


0.0 


100.0 


0.0 


0.0 


5.9 


8.2 


11.6 


11.9 


9.7 


0.0 


0.0 


0.0 


0.0 


0.0 100.0 


0.0 


2.1 


3.8 


8.8 


23.4 


7.5 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


100.0 



S9, 



BEST COPY AVAIUBLE 



EXHIBIT B-3a 

Estimat.d Population Distributions of National Assessment 
Reporting Groups (AH 17-Year-0Id8) 



Variables Natl. Region (%) Sex (it) Color (%) 

and Groups (%} 

SE W C NE M F B W 0th. 



Region 
Southeast 
West 
Central 
Northeast 



21.6 


100.0 


0.0 


0.0 


0.0 


21.1 


22.2 


42.9 


20.1 


0.8 


24.9 


0.0 


100.0 


0.0 


0.0 


24.9 


25.0 


18.5 


22.2 


80.6 


28.2 


0.0 


0.0 


100.0 


0.0 


28.7 


27.8 


21.9 


30.6 


4.8 


25.2 


0.0 


0.0 


0.0 


100.0 


25.4 


25.1 


16.6 


27.1 


13.9 



Sex 
Male 
Female 



48.7 
51.3 



47.4 
52.6 



48.6 
51.4 



49.5 
50.5 



49.1 
51.0 



100.0 
0.0 



0.0 
100.0 



46.1 
53.9 



49.0 
51.0 



49.3 
50.7 



Color 
Black 
White 
Other 



11.3 
83.3 
5.4 



22.3 
77.5 
0.2 



8.4 
74.2 
17.5 



8.8 
90.3 
0.9 



7.4 
89.6 
3.0 



10.7 
83.9 
5.5 



11.8 
82.8 
5.3 



100.0 
0.0 
0.0 



0.0 
100.0 
0.0 



0.0 
0.0 
100.0 



Parental education 



No high school 


8.4 


12.3 


9.7 


6.8 


5.7 


8.2 


8.7 


15.1 


6.1 


29.8 


Some high school 


12.6 


16.9 


11.8 


12.2 


10.1 


11.3 


13.9 


22.7 


10.9 


17.4 


Graduated high school 


30.3 


28.5 


25.7 


35.2 


30.8 


30.9 


29.7 


27.8 


31.4 


18.8 


Post high school 


43.6 


38.4 


46.5 


42.2 


46.9 


44.0 


43.3 


23.8 


47.8 


20.5 


Unknown 


5.1 


3.8 


6.3 


3.6 


6.5 


5.6 


4.5 


10.6 


3.8 


13.5 



Size and type of 
community (STOC) 
Low metro 
Extreme rural 
Small place 
Medium city 
Main big city 
Urban fringe 
High metro 



10.7 


7.1 


14.1 


10.9 


10.2 


10.7 


10.7 


38.0 


5.9 


28.4 


9.2 


11.3 


6.1 


15.5 


3.3 


9.2 


9.1 


6.6 


9.4 


10.5 


31.0 


36.2 


29.4 


30.7 


28.3 


30.5 


31.4 


19.5 


33.2 


20.7 


16.5 


23.2 


15.6 


15.3 


13.0 


16.5 


16.5 


16.9 


16.7 


9.2 


9.3 


8.5 


11.0 


7.2 


10.8 


8.9 


9.8 


12.4 


8.3 


18.4 


7.2 


3.5 


9.9 


7.1 


7.8 


6.8 


7.5 


1.6 


8.0 


6.2 


16.2 


10.3 


14.1 


13.3 


26.6 


17.4 


15.0 


5.2 


18.3 


6.6 



BEST COPY mmii 



EXHIBIT B-3b 

Estimated Population Distributions of National Assessment 
Reporting Groups (All 17** Year-Olds) 



Variabl.'^s 
and Groups 



Parental Education C%) 



Size and Type of Comnunity (%) 





NHS 


SHS 


GHS 


FHS 


Unk. 


LM 


ER 


SP 


MC 


MBC 


UF 


HM 


Regicn 


























Southeast 


31.5 


29.0 


20.4 


19.1 


16.4 


14.3 


26.7 


25.3 


30.4 


19.6 


10.4 


13.7 


West 


28.8 


23.3 


21.1 


26.6 


30.9 


32.8 


16.5 


23.7 


23.6 


29.3 


34.3 


21.7 


Central 


22.6 


27.3 


32.8 


27.3 


20.3 


28.8 


47.7 


28.0 


26.1 


21.8 


27.9 


23.2 


Northeast 


17.1 


20.3 


25.7 


27.1 


32.4 


24.1 


9.2 


23.0 


19.9 


29.3 


27.4 


41.4 



Sex 
Male 
Fetule 



47.4 43.6 49.7 49.1 54.5 
52.6 56.4 50.3 50.9 45.5 



48.8 49.0 48.0 48.7 46.3 46.3 52.4 
51.2 51.0 52.0 51.3 53.7 53.7 47.6 



Color 
Black 
White 
Other 



20.2 20.3 
60.7 72.3 
19.1 7.5 



10.3 
86.3 
3.4 



6.1 
91.3 
2.5 



23.7 
62.0 
14.4 



40.0 
45.7 
14.4 



8.1 
85.7 
6.2 



7.1 
89.3 
3.6 



11.4 
85.6 
3.0 



15.0 
74.3 
10.6 



2.5 3.6 
92.8 94.2 
4.7 2.2 



Parental education 



No high school 


100.0 


0.0 


0.0 


0.0 


0.0 


10.3 


14.9 


9.9 


7.4 


7.5 


8.4 


2.4 


Some high school 


0.0 


100.0 


0.0 


0.0 


0.0 


16.4 


15.6 


15.1 


12.2 


12.5 


11.2 


4.7 


Graduated high school 


0.0 


0.0 


100.0 


0.0 


0.0 


30.9 


35.4 


33.1 


31.1 


28.8 


28.7 


22.1 


Post high school 


0.0 


0.0 


0.0 


100.0 


0.0 


32.9 


29.4 


38.6 


45.5 


36.2 


48.3 


68.8 


Unknown 


0.0 


0.0 


0.0 


0.0 


100.0 


9.5 


4.7 


3.3 


3.8 


14.9 


3.5 


2.0 



Size and type of 
coamunity (STOC) 
Low metro 
Extreme rural 
Small place 
Medium city 
Main big city 
Urban fringe 
High metro 



13.0 


13.9 


10.9 


8.1 


20.2 


100.0 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


16.2 


11.3 


10.7 


6.2 


8.6 


0.0 


100.0 


0.0 


0.0 


0.0 


0.0 


0.0 


36.3 


37.1 


33.9 


27.4 


20.0 


0.0 


0.0 100. 0 


0.0 


0.0 


0.0 


0.0 


14.5 


15.9 


16.9 


17.2 


12.3 


0.0 


0.0 


0.0 100.0 


0.0 


0.0 


0.0 


8.3 


9.3 


8.9 


7.7 


27.5 


0.0 


0.0 


0.0 


0.0 


100.0 


0.0 


0.0 


7.1 


6.4 


6.8 


8.0 


5.0 


0.0 


0.0 


0.0 


0.0 


0.0 


100.0 


0.0 


4.5 


6.1 


11.8 


25.5 


6.4 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 100.0 



EXHIBIT B-4a 



BEST COPY AVAILABLE 



Estimated Population Distributions of National Assessment 
Reporting Croups (Young Adults) 



l:itl:is V)' '-""^^ -iS^LjSI. Color 

SE W C NE M F B W 0th. 



Region 

Southeast 21.2 100.0 0.0 0.0 0.0 21.2 21.2 34.7 20.6 



4.6 



J!*" , 25.5 0.0 100.0 0.0 0.0 25.7 25.3 15.8 23.7 71.4 

Central 29.0 0.0 0.0 100.0 0.0 29.1 29.0 22.4 31.4 5.6 

Northeast 24.2 0.0 0.0 0.0 100.0 23.9 24.5 27.1 24.3 18.3 



Sex 

*9-0 49.0 49.5 49.1 48.4 100.0 0.0 45.9 49.1 53.6 
51.0 51.0 50.5 50.9 51.6 0.0 100.0 54.2 50.9 46.4 



Color 

^^<^^ 10.7 17.4 6.6 8.3 12.0 10.0 11.4 100.0 



^^^^ 83.7 81.3 77.8 90.7 83.8 83.9 83.6 0.0 100.0 

5.6 1.2 15.6 1.1 4.2 6.1 5.1 0.0 0.0 100.0 



0.0 0.0 
0.0 



Parental education 
No high school 
Some high school 
Graduated high school 
Post high school 
Unknown 



28.4 


36.3 


26.0 


28.1 


24.5 


27.7 


29.1 


41.8 


24.4 


63.4 


15.9 


16.5 


13.1 


17.3 


16.7 


15.1 


16.7 


22.6 


15.6 


8.6 


29.8 


23.2 


32.1 


30.4 


32.6 


31.5 


28.2 - 


21.4 


31.9 


15.0 


22.4 


18.9 


27.2 


20.2 


23.2 


22.5 


22.4 


7.2 


25.2 


9.8 


3.4 


5.1 


1.6 


4.0 


3.0 


3.2 


3.5 


7.0 


2.9 


3.2 



Size and type of 
community (STOC) 
Low metro 
Extreme rural 
Small place 
Medium city 
Main big city 
Urban fringe 
Hii^h metro 



9.0 


3.5 


11.3 


9.0 


11.4 


9.1 


9.0 


42.1 


3.9 


22.8 


7.6 


7.7 


6.3 


14.7 


0.5 


7.6 


7.7 


6.2 


7.7 


9.8 


30.4 


43.2 


29.7 


27.0 


24.0 


30.8 


30.0 


13.5 


33.6 


14.8 


12.8 


26.5 


10.3 


10.3 


6.4 


13.1 


12.5 


16.4 


12.8 


6.4 


10.5 


2.0 


16.8 


8.6 


13.6 


10.6 


10.4 


12.0 


8.9 


31.2 


17.5 


6.9 


13.8 


20.9 


26.7 


17.7 


17.3 


6.8 


19.6 


7.4 


12.2 


10.2 


11.9 


9.5 


17.4 


11.2 


13.2 


3.1 


13.6 


7.8 



BEST copy mmu 



EXHIBIT B-4b 

Estimated Population Distributions of National Assessment 
Reporting Groups (Young Adults) 



Var iables Parental Education (!!!) Size and Type of Community (%) 

and Groups 

NHS SHS GHS PHS Unk. LM ER SP MC MBC UF UM 



Region 

Southeast 27.1 22.0 16.5 17.9 32.1 8.4 21.5 30.2 44.0 4.1 8.3 17.7 

West 23.3 21.0 27.5 30.9 12.1 32.0 20.9 24.9 20.5 40.8 20.1 24.9 

Central 28.7 31.5 29.6 26.2 34.3 29.1 55.9 25.8 23.4 23.8 34.7 22.7 

Northeast 20.9 25*5 26.4 25.1 21.5 30.6 1.7 19.1 12.1 31.3 36.9 34.7 



Sex 

Male 47.8 46.4 51.8 49.1 46.9 49.3 48.6 49.7 50.3 49.5 49.6 44.9 

Female 52.2 53.6 48.2 50.9 53.1 50.7 51.4 50.3 49.7 50.5 50.4 55.1 



Color 

Black 15.7 15.2 7.7 3.5 22.1 49.9 8.7 4.7 13.7 12.2 4.1 2.7 

White 71.9 81.8 89.5 94.1 72.7 36.0 84.2 92.6 83.6 71.2 93.5 93.7 

Other 12.4 3.0 2.8 2.4 5.3 14.1 7.1 2.7 2.8 16.6 2.4 3.6 



Parental education 



No high school 


100.0 


0.0 


0.0 


0.0 


0.0 


37.0 


40.9 


28.9 


31.0 


33.3 


25.0 


11.4 


Some high school 


0.0 


100.0 


0.0 


0.0 


0.0 


24.7 


13.8 


14.7 


17.3 


17.4 


16.4 


10.3 


Graduated high school 


0.0 


0.0 


100.0 


0.0 


0.0 


24.2 


25.5 


29.8 


27.9 


26.5 


35.4 


33.7 


Post high school 


0.0 


0.0 


0*0 


100.0 


0.0 


8.5 


16.2 


21.8 


20.7 


21.5 


20.1 


44.2 


Unknown 


0.0 


0.0 


0.0 


0.0 


100.0 


5.5 


3.6 


4.8 


3.2 


1.3 


3.2 


0.4 



Size and type of 
community (STOC) 



Low metro 


11.7 


14.0 


7.3 


3.4 


14.7 


100.0 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


Extreme rural 


11.0 


6.6 


6.5 


5.5 


8.2 


0.0 


100.0 


0.0 


0.0 


0.0 


0.0 


0.0 


Small place 


30.9 


28.1 


30.4 


29.6 


42.8 


0.0 


0.0 


100.0 


0.0 


0.0 


u.o 


0.0 


Medium city 


13.9 


13.9 


12.0 


11.8 


12.2 


0.0 


0.0 


0.0 


100.0 


0.0 


0.0 


0.0 


Main big city 


12.3 


11.5 


9.4 


10.1 


4.1 


0.0 


0.0 


0.0 


0.0 


100.0 


0.0 


0.0 


Urban fringe 


15.4 


18.1 


20.7 


15.7 


16.6 


0.0 


0.0 


0.0 


0.0 


0.0 


100.0 


0.0 


High metro 


4.9 


7.9 


13.8 


24.0 


1.4 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


100.0 



54i 



BEST COPY AVAILABLE 



APPENDIX C 



NATIONAL ASSESSMENT POLICY COMMITTEE 



(Chairman of Policy Committee) 
Mr. Robert McBride, President-Elect 
Nat'l. Assoc. of School Boards of 
Education 

Dr. George Brain 
Dean, College of Education 
Washington State University 
Pullman, Washington 

Mr. William R. Conway 

Education Committee 

Florida House of Representatives 

Dr. Lyman V. Ginger 
Superintendent of Public Instruction 
State Department of Education 
Frankfort, Kentucky 

Dr. George Kozmetsky 

Dean, Graduate School of Business 

University of Texas 



Ms. Joyce E. Lewis 
Maine House of Representatives 
Education Committee 



Dr. Bill Lillard 
Superintendent of Schools 
Oklahoma City, Oklahoma 



Dr. Frederick Mosteller 
Chairman, Department of Statistics 
Harvard University 

Ms. Eleanor P. Sheppard 
Chairman, Education Committee 
Virginia House of Representatives 



Dr. Stephen Wright 
Vice President, College Entrance 
Examination Board 



ex officio 
Dr. Ralph W. Tyler 
Science Research Associates 
Chicago, Illinois 



BEST COPY AVAIUBLE 



APPENDIX D 

NATIONAL ASSESSMENT ANALYSIS ADVISORY COMMITTEE 

Dr. Frederick Mosteller, Chairman since 1973. Professor of Mathematical Statistics, Harvard 
University (1970— ). 

Dr. John W. Tukey , Chairman 1965-73. Professor of Statistics, Princeton University (1965- ). 

Dr. Robert Abelson, Professor of Psychology, University of North Carolina (1965-72). 

Dr. David Brillinger, Professor of Statistics, University of California at Berkeley (1973- ). 

Dr. William E. Coffman, Professor of Education, University of Iowa (1970- ). 

Dr. Lee J. Cronbach, Professor of Psychology and Educat:on, Stanford University (1965-69). 

Dr. James A. Davis, Director of National Opinion Research Center, Chicago, Illinois ( 1973— ). 

Dr. Janet Elashoff , Statistical Advisor, Center for Advanced Study in the Behavioral Sciences, Palo 
Alto, California (1973- ). 

Dr. John P. Gilbert, Staff Statistician at the Harvard University Computer Center (1970- ). 

Dr. Lyle V. Jones, Professor of Psychology, L.L. Thurstone Psychometric Laboratory, Dean of the 
Graduate School, University of North Carolina ( 1965- ). 

Dr. Lincoln Moses, Professor of Statistics, Dean of the Graduate School, Stanford University 
(1974- ). 

Dr. Ralph W. Tyler, Science Research Associates, Chicago, Illinois (1965-69). 



S«6 

o 

ERIC 



